[PATCH v3 0/6] Fixes usage of additional apt keys and repos

[PATCH v3 0/6] Fixes usage of additional apt keys and repos

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Diff to v2:
	* split patches in a clean way:
	* only install https-support and ca-certificates if
	  we have provided a custom apt-key
	  (see https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009)
	  - use the existing override mechanism for this
	* first improve existing isar code before adding my own
	* don't include "tmp/" inside of $kfn

What this series does:

Enable the user to really ADD bootstrap repos and keys without
replacing existing ones.

The existing keyring is not replaced but keys are added to
/ect/apt/trusted.gpg.d/isar.gpg instead, where debootstrap and any apt
call can find it.

Furthermore, the code to add keys is simplified by removing duplicate
code and not manually parsing URIs and guessing about download locations
as well as not manually handling gpg and giving apt config overrides.

It is much simpler by using `apt-key` and default apt keyring paths.

Furthermore, apt-get must not use a given single source list which was
used from debootstrapping. Otherwise, additional packages are always
unauthenticated, which is a quite misleading error. Instead, apt-get
should use all source lists available in the built root.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>

Andreas Reichel (6):
  Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"
  Remove duplicate code from apt-keyring generation
  Fix key filename in apt-keyring generator
  Use apt-key to generate apt-keyring
  Use all source lists in target root apt
  If we use a custom keyring debootstrap may fall to https

 .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
 .../conf/multiconfig/qemuamd64-jessie.conf    |  1 -
 meta/classes/isar-bootstrap-helper.bbclass    | 17 +++++--
 meta/classes/isar-image.bbclass               |  1 +
 .../isar-bootstrap/isar-bootstrap-host.bb     |  2 +-
 .../isar-bootstrap/isar-bootstrap.inc         | 49 +++++++++++--------
 6 files changed, 43 insertions(+), 28 deletions(-)

-- 
2.21.0

[PATCH v3 1/6] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.

We do not want to guess where the downloaded file will be. This does
not work anymore if the key is downloaded from remote with a URL.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 234d339..eb9a9df 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -40,7 +40,7 @@ python () {
         d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
         for key in distro_apt_keys.split():
             url = urlparse(key)
-            filename = ''.join([wd, url.path])
+            filename = os.path.basename(url.path)
             d.appendVar("SRC_URI", " %s" % key)
             d.appendVar("APTKEYFILES", " %s" % filename)
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
-- 
2.21.0

[PATCH v3 2/6] Remove duplicate code from apt-keyring generation

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 .../isar-bootstrap/isar-bootstrap.inc         | 24 +++++++++----------
 1 file changed, 11 insertions(+), 13 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index eb9a9df..204ed1f 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -35,23 +35,21 @@ inherit base-apt-helper
 python () {
     from urllib.parse import urlparse
     distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
-    wd = d.getVar("WORKDIR", True)
+    aptkeys = []
+
     if distro_apt_keys:
-        d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
-        for key in distro_apt_keys.split():
-            url = urlparse(key)
-            filename = os.path.basename(url.path)
-            d.appendVar("SRC_URI", " %s" % key)
-            d.appendVar("APTKEYFILES", " %s" % filename)
+        aptkeys += distro_apt_keys.split()
     if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
         own_pub_key = d.getVar("BASE_REPO_KEY", False)
         if own_pub_key:
-            d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
-            for key in own_pub_key.split():
-                url = urlparse(key)
-                filename = ''.join([wd, url.path])
-                d.appendVar("SRC_URI", " %s" % key)
-                d.appendVar("APTKEYFILES", " %s" % filename)
+            aptkeys += own_pub_key.split()
+
+    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
+    for key in aptkeys:
+        url = urlparse(key)
+        filename = os.path.basename(url.path)
+        d.appendVar("SRC_URI", " %s" % key)
+        d.appendVar("APTKEYFILES", " %s" % filename)
 }
 
 def aggregate_files(d, file_list, file_out):
-- 
2.21.0

[PATCH v3 3/6] Fix key filename in apt-keyring generator

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Use bb.fetch2.Fetch to retrieve the final filename after it is
downloaded. This way we don't have to guess (wrongly), and also
additional SRC_URI parameters like subdir or filename are usable now.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 5 ++---
 1 file changed, 2 insertions(+), 3 deletions(-)

diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 204ed1f..98c5c6b 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -33,7 +33,6 @@ DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BA
 inherit base-apt-helper
 
 python () {
-    from urllib.parse import urlparse
     distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
     aptkeys = []
 
@@ -46,9 +45,9 @@ python () {
 
     d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
     for key in aptkeys:
-        url = urlparse(key)
-        filename = os.path.basename(url.path)
         d.appendVar("SRC_URI", " %s" % key)
+        fetcher = bb.fetch2.Fetch([key], d)
+        filename = fetcher.localpath(key)
         d.appendVar("APTKEYFILES", " %s" % filename)
 }
 
-- 
2.21.0

[PATCH v3 4/6] Use apt-key to generate apt-keyring

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

Use apt-key instead of manually calling gpg.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/classes/isar-bootstrap-helper.bbclass        | 12 ++++++++++++
 meta/classes/isar-image.bbclass                   |  1 +
 .../isar-bootstrap/isar-bootstrap-host.bb         |  2 +-
 .../isar-bootstrap/isar-bootstrap.inc             | 15 +++++++--------
 4 files changed, 21 insertions(+), 9 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index d780b85..1f24037 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -22,6 +22,7 @@ HOST_DISTRO ?= "debian-stretch"
 HOST_ARCH ?= "${@get_deb_host_arch()}"
 
 HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
+ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
 
 def reverse_bb_array(d, varname):
     array = d.getVar(varname, True)
@@ -119,6 +120,16 @@ setup_root_file_system() {
     export LANG=C
     export LANGUAGE=C
     export LC_ALL=C
+
+    if [ -d ${TMPDIR}/aptkeys ]; then
+        for keyfile in ${TMPDIR}/aptkeys/*
+        do
+            kfn="$(basename $keyfile)"
+            cp $keyfile "$ROOTFSDIR/tmp/$kfn"
+            sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add "/tmp/$kfn"
+            rm "$ROOTFSDIR/tmp/$kfn"
+        done
+    fi
     sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
         -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
         -o Dir::Etc::sourceparts="-" \
@@ -128,6 +139,7 @@ setup_root_file_system() {
         sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
         sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
     fi
+    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
     sudo -E chroot "$ROOTFSDIR" \
         /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
             ${IMAGE_TRANSIENT_PACKAGES}
diff --git a/meta/classes/isar-image.bbclass b/meta/classes/isar-image.bbclass
index cdd1651..4a89bd7 100644
--- a/meta/classes/isar-image.bbclass
+++ b/meta/classes/isar-image.bbclass
@@ -82,6 +82,7 @@ isar_image_cleanup() {
         fi
         rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
     '
+    sudo rm -f "${ISARKEYRING}"
 }
 
 do_rootfs() {
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index a793585..b70d2a8 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -11,8 +11,8 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
 DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
 ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
 
-require isar-bootstrap.inc
 inherit isar-bootstrap-helper
+require isar-bootstrap.inc
 
 do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
 
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 98c5c6b..8002a53 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -23,10 +23,8 @@ APTSRCS = "${WORKDIR}/apt-sources"
 APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
 BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
 APTKEYFILES = ""
-APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
-DEBOOTSTRAP_KEYRING = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2"
 
 DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
@@ -43,7 +41,6 @@ python () {
         if own_pub_key:
             aptkeys += own_pub_key.split()
 
-    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
     for key in aptkeys:
         d.appendVar("SRC_URI", " %s" % key)
         fetcher = bb.fetch2.Fetch([key], d)
@@ -171,13 +168,17 @@ def get_distro_components_argument(d, is_host):
     else:
         return ""
 
+APTKEYTMPDIR := "${TMPDIR}/aptkeys"
+
+do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
 do_generate_keyring[dirs] = "${DL_DIR}"
 do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
 do_generate_keyring() {
     if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
+        chmod 777 "${APTKEYTMPDIR}"
         for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
-           gpg --no-default-keyring --keyring "${APTKEYRING}" \
-               --no-tty --homedir "${DL_DIR}"  --import "$keyfile"
+           cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
+           sudo apt-key --keyring "${ISARKEYRING}" add "$keyfile"
         done
     fi
 }
@@ -219,7 +220,6 @@ isar_bootstrap() {
             if [ ${IS_HOST} ]; then
                 ${DEBOOTSTRAP} $debootstrap_args \
                                ${@get_distro_components_argument(d, True)} \
-                               ${DEBOOTSTRAP_KEYRING} \
                                "${@get_distro_suite(d, True)}" \
                                "${ROOTFSDIR}" \
                                "${@get_distro_source(d, True)}"
@@ -228,7 +228,6 @@ isar_bootstrap() {
                  "${DEBOOTSTRAP}" $debootstrap_args \
                                   --arch="${DISTRO_ARCH}" \
                                   ${@get_distro_components_argument(d, False)} \
-                                  ${DEBOOTSTRAP_KEYRING} \
                                   "${@get_distro_suite(d, False)}" \
                                   "${ROOTFSDIR}" \
                                   "${@get_distro_source(d, False)}"
-- 
2.21.0

[PATCH v3 5/6] Use all source lists in target root apt

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

When we only use isar-apt.list, we cannot add additional repositories
since they are listed in the bootstrap list only.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta/classes/isar-bootstrap-helper.bbclass | 5 +----
 1 file changed, 1 insertion(+), 4 deletions(-)

diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 1f24037..f9e3fba 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -130,10 +130,7 @@ setup_root_file_system() {
             rm "$ROOTFSDIR/tmp/$kfn"
         done
     fi
-    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
-        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
-        -o Dir::Etc::sourceparts="-" \
-        -o APT::Get::List-Cleanup="0"
+    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
     # Add multiarch for cross-target
     if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then
         sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
-- 
2.21.0

[PATCH v3 6/6] If we use a custom keyring debootstrap may fall to https

[Andreas J. Reichel]

From: Andreas Reichel <andreas.reichel.ext@siemens.com>

See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009

So if we have something in aptkeyring, append https-support to
OVERRIDES.

Furthermore, the conditional append for https-support was missing
in qemuamd64-stretch.conf, thus, remove this from all the distros
and put it into the isar-bootstrap.inc.

Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
 meta-isar/conf/multiconfig/qemuamd64-buster.conf    |  1 -
 meta-isar/conf/multiconfig/qemuamd64-jessie.conf    |  1 -
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 11 +++++++++++
 3 files changed, 11 insertions(+), 2 deletions(-)

diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf b/meta-isar/conf/multiconfig/qemuamd64-buster.conf
index 63df75c..da90993 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-buster.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-buster.conf
@@ -18,4 +18,3 @@ QEMU_MACHINE ?= "q35"
 QEMU_CPU ?= ""
 QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE## -bios /usr/local/share/ovmf/OVMF.fd"
 
-DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates"
diff --git a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
index d1335ff..42c71df 100644
--- a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
+++ b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
@@ -15,4 +15,3 @@ QEMU_MACHINE ?= "pc"
 QEMU_CPU ?= ""
 QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##"
 
-DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 8002a53..64cefc6 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -25,6 +25,7 @@ BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
 APTKEYFILES = ""
 DEPLOY_ISAR_BOOTSTRAP ?= ""
 DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2"
+DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = " apt-transport-https ca-certificates"
 
 DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org  file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
 
@@ -41,6 +42,12 @@ python () {
         if own_pub_key:
             aptkeys += own_pub_key.split()
 
+    if len(aptkeys) > 0:
+        # debootstrap falls back to https if there is no
+        # 'reliable' keyring, whatever that means, but it happened
+        # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
+        d.setVar("HAVE_CUSTOM_APT_KEYS", "True")
+
     for key in aptkeys:
         d.appendVar("SRC_URI", " %s" % key)
         fetcher = bb.fetch2.Fetch([key], d)
@@ -150,6 +157,10 @@ def get_distro_have_https_source(d, is_host=False):
     return any(source[2].startswith("https://") for source in generate_distro_sources(d, is_host))
 
 def get_distro_needs_https_support(d, is_host=False):
+    apt_keys = d.getVar("HAVE_CUSTOM_APT_KEYS", False)
+    if apt_keys:
+        return "https-support"
+
     if get_distro_have_https_source(d, is_host):
         return "https-support"
     else:
-- 
2.21.0

Re: [PATCH v3 1/6] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Henning Schild]

I do not see the point of the revert. You are just going back to the way
it was broken before. It also does not seem to document what went wrong.

Instead of that revert 2/6 needs a more verbose commit message maybe
mentioning the previous patches that got it wrong.

Henning

Am Wed, 6 Mar 2019 17:26:13 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
> 
> We do not want to guess where the downloaded file will be. This does
> not work anymore if the key is downloaded from remote with a URL.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
>  1 file changed, 1 insertion(+), 1 deletion(-)
> 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> 234d339..eb9a9df 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -40,7 +40,7
> @@ python () { d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> ${APTKEYRING}") for key in distro_apt_keys.split():
>              url = urlparse(key)
> -            filename = ''.join([wd, url.path])
> +            filename = os.path.basename(url.path)
>              d.appendVar("SRC_URI", " %s" % key)
>              d.appendVar("APTKEYFILES", " %s" % filename)
>      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):

Re: [PATCH v3 5/6] Use all source lists in target root apt

[Henning Schild]

Am Wed, 6 Mar 2019 17:26:17 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> When we only use isar-apt.list, we cannot add additional repositories
> since they are listed in the bootstrap list only.

As far as i understood some entries in the list are not "enabled" or
fully working before we added the key. The reasoning behind this change
is somehow gpg ... and that needs to go in here. Because that is the
first violation of "we only update isar-apt". But the violation is so
early in the bootstrap that it is probably ok.

Henning

> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta/classes/isar-bootstrap-helper.bbclass | 5 +----
>  1 file changed, 1 insertion(+), 4 deletions(-)
> 
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index 1f24037..f9e3fba
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -130,10 +130,7 @@ setup_root_file_system() {
>              rm "$ROOTFSDIR/tmp/$kfn"
>          done
>      fi
> -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> -        -o Dir::Etc::sourceparts="-" \
> -        -o APT::Get::List-Cleanup="0"
> +    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
>      # Add multiarch for cross-target
>      if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then
>          sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture
> ${DISTRO_ARCH}

Re: [PATCH v3 4/6] Use apt-key to generate apt-keyring

[Henning Schild]

Am Wed, 6 Mar 2019 17:26:16 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Use apt-key instead of manually calling gpg.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta/classes/isar-bootstrap-helper.bbclass        | 12 ++++++++++++
>  meta/classes/isar-image.bbclass                   |  1 +
>  .../isar-bootstrap/isar-bootstrap-host.bb         |  2 +-
>  .../isar-bootstrap/isar-bootstrap.inc             | 15
> +++++++-------- 4 files changed, 21 insertions(+), 9 deletions(-)
> 
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index d780b85..1f24037
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -22,6 +22,7 @@ HOST_DISTRO ?= "debian-stretch"
>  HOST_ARCH ?= "${@get_deb_host_arch()}"
>  
>  HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
> +ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
>  
>  def reverse_bb_array(d, varname):
>      array = d.getVar(varname, True)
> @@ -119,6 +120,16 @@ setup_root_file_system() {
>      export LANG=C
>      export LANGUAGE=C
>      export LC_ALL=C
> +
> +    if [ -d ${TMPDIR}/aptkeys ]; then
> +        for keyfile in ${TMPDIR}/aptkeys/*
> +        do
> +            kfn="$(basename $keyfile)"
> +            cp $keyfile "$ROOTFSDIR/tmp/$kfn"
> +            sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> "/tmp/$kfn"
> +            rm "$ROOTFSDIR/tmp/$kfn"
> +        done
> +    fi
>      sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
>          -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
>          -o Dir::Etc::sourceparts="-" \
> @@ -128,6 +139,7 @@ setup_root_file_system() {
>          sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture
> ${DISTRO_ARCH} sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
>      fi
> +    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
>      sudo -E chroot "$ROOTFSDIR" \
>          /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
>              ${IMAGE_TRANSIENT_PACKAGES}
> diff --git a/meta/classes/isar-image.bbclass
> b/meta/classes/isar-image.bbclass index cdd1651..4a89bd7 100644
> --- a/meta/classes/isar-image.bbclass
> +++ b/meta/classes/isar-image.bbclass
> @@ -82,6 +82,7 @@ isar_image_cleanup() {
>          fi
>          rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
>      '
> +    sudo rm -f "${ISARKEYRING}"
>  }
>  
>  do_rootfs() {
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index
> a793585..b70d2a8 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -11,8
> +11,8 @@ WORKDIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
> DEPLOY_ISAR_BOOTSTRAP =
> "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
> ISAR_BOOTSTRAP_LOCK =
> "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock" -require
> isar-bootstrap.inc inherit isar-bootstrap-helper +require
> isar-bootstrap.inc do_generate_keyring[stamp-extra-info] =
> "${DISTRO}-${DISTRO_ARCH}" 
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> 98c5c6b..8002a53 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,10 +23,8
> @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> -DEBOOTSTRAP_KEYRING = ""
>  DEPLOY_ISAR_BOOTSTRAP ?= ""
> -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2"

When i talked about the OVERRIDES i meant also adding gnupg2 depending
on a condition. And that condition would probably be, if
DISTRO_APT_KEYS is non-empty.

DISTRO_APT_KEYS != "" && OVERRIDES += "gnupg-support"
DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg-support = " gnupg2"
 
>  DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> file:///${REPO_BASE_DIR} \n" if
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> @@ -43,7 +41,6 @@ python () { if own_pub_key:
>              aptkeys += own_pub_key.split()
>  
> -    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
>      for key in aptkeys:
>          d.appendVar("SRC_URI", " %s" % key)
>          fetcher = bb.fetch2.Fetch([key], d)
> @@ -171,13 +168,17 @@ def get_distro_components_argument(d, is_host):
>      else:
>          return ""
>  
> +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> +
> +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
>  do_generate_keyring[dirs] = "${DL_DIR}"
>  do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
>  do_generate_keyring() {
>      if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> +        chmod 777 "${APTKEYTMPDIR}"
>          for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> -           gpg --no-default-keyring --keyring "${APTKEYRING}" \
> -               --no-tty --homedir "${DL_DIR}"  --import "$keyfile"
> +           cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> +           sudo apt-key --keyring "${ISARKEYRING}" add "$keyfile"

This whole switching from gpg to "sudo chroot apt-key" would make a
nice stand-alone patch, but i do not feel strong about that.

Henning

>          done
>      fi
>  }
> @@ -219,7 +220,6 @@ isar_bootstrap() {
>              if [ ${IS_HOST} ]; then
>                  ${DEBOOTSTRAP} $debootstrap_args \
>                                 ${@get_distro_components_argument(d,
> True)} \
> -                               ${DEBOOTSTRAP_KEYRING} \
>                                 "${@get_distro_suite(d, True)}" \
>                                 "${ROOTFSDIR}" \
>                                 "${@get_distro_source(d, True)}"
> @@ -228,7 +228,6 @@ isar_bootstrap() {
>                   "${DEBOOTSTRAP}" $debootstrap_args \
>                                    --arch="${DISTRO_ARCH}" \
>                                    ${@get_distro_components_argument(d,
> False)} \
> -                                  ${DEBOOTSTRAP_KEYRING} \
>                                    "${@get_distro_suite(d, False)}" \
>                                    "${ROOTFSDIR}" \
>                                    "${@get_distro_source(d, False)}"

Re: [PATCH v3 6/6] If we use a custom keyring debootstrap may fall to https

[Henning Schild]

Am Wed, 6 Mar 2019 17:26:18 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:

> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> 
> So if we have something in aptkeyring, append https-support to
> OVERRIDES.
> 
> Furthermore, the conditional append for https-support was missing
> in qemuamd64-stretch.conf, thus, remove this from all the distros
> and put it into the isar-bootstrap.inc.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
>  meta-isar/conf/multiconfig/qemuamd64-buster.conf    |  1 -
>  meta-isar/conf/multiconfig/qemuamd64-jessie.conf    |  1 -
>  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 11 +++++++++++
>  3 files changed, 11 insertions(+), 2 deletions(-)
> 
> diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf
> b/meta-isar/conf/multiconfig/qemuamd64-buster.conf index
> 63df75c..da90993 100644 ---
> a/meta-isar/conf/multiconfig/qemuamd64-buster.conf +++
> b/meta-isar/conf/multiconfig/qemuamd64-buster.conf @@ -18,4 +18,3 @@
> QEMU_MACHINE ?= "q35" QEMU_CPU ?= ""
>  QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##
> -bios /usr/local/share/ovmf/OVMF.fd" 
> -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> apt-transport-https ca-certificates" diff --git
> a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
> b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf index
> d1335ff..42c71df 100644 ---
> a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf +++
> b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf @@ -15,4 +15,3 @@
> QEMU_MACHINE ?= "pc" QEMU_CPU ?= "" QEMU_DISK_ARGS ?= "-hda
> ##ROOTFS_IMAGE##" 
> -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> apt-transport-https ca-certificates" diff --git
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> 8002a53..64cefc6 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -25,6 +25,7
> @@ BASEAPTSRCS = "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales
> gnupg2" +DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> apt-transport-https ca-certificates" 
>  DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> file:///${REPO_BASE_DIR} \n" if
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> @@ -41,6 +42,12 @@ python () { if own_pub_key:
>              aptkeys += own_pub_key.split()
>  
> +    if len(aptkeys) > 0:
> +        # debootstrap falls back to https if there is no
> +        # 'reliable' keyring, whatever that means, but it happened
> +        # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> +        d.setVar("HAVE_CUSTOM_APT_KEYS", "True")

Why this indirection and the new variable? Can you not just check
len(aptkeys) where you check HAVE_CUSTOM_APT_KEYS?

In fact you probably want to look at DISTRO_APT_KEYS, which seems to be
the source feeding all the other variables. Whatever you come up with
in v3 4/6.

>      for key in aptkeys:
>          d.appendVar("SRC_URI", " %s" % key)
>          fetcher = bb.fetch2.Fetch([key], d)
> @@ -150,6 +157,10 @@ def get_distro_have_https_source(d,
> is_host=False): return any(source[2].startswith("https://") for
> source in generate_distro_sources(d, is_host)) 
>  def get_distro_needs_https_support(d, is_host=False):
> +    apt_keys = d.getVar("HAVE_CUSTOM_APT_KEYS", False)
> +    if apt_keys:
> +        return "https-support"
> +

Reusing the OVERRIDE seems like a good idea to avoid double adding the
packages. So gnupg implies https but does not add the packages again.
But i think i would do something like

get_gnupg_overrides() 
 ret = "gnupg"
 # blabla bug
 ret += " https-support"

Henning

>      if get_distro_have_https_source(d, is_host):
>          return "https-support"
>      else:

Re: [PATCH v3 4/6] Use apt-key to generate apt-keyring

[Andreas Reichel]

On Wed, Mar 06, 2019 at 06:33:12PM +0100, Henning Schild wrote:
> Am Wed, 6 Mar 2019 17:26:16 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> 
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > Use apt-key instead of manually calling gpg.
> > 
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> >  meta/classes/isar-bootstrap-helper.bbclass        | 12 ++++++++++++
> >  meta/classes/isar-image.bbclass                   |  1 +
> >  .../isar-bootstrap/isar-bootstrap-host.bb         |  2 +-
> >  .../isar-bootstrap/isar-bootstrap.inc             | 15
> > +++++++-------- 4 files changed, 21 insertions(+), 9 deletions(-)
> > 
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index d780b85..1f24037
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -22,6 +22,7 @@ HOST_DISTRO ?= "debian-stretch"
> >  HOST_ARCH ?= "${@get_deb_host_arch()}"
> >  
> >  HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
> > +ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
> >  
> >  def reverse_bb_array(d, varname):
> >      array = d.getVar(varname, True)
> > @@ -119,6 +120,16 @@ setup_root_file_system() {
> >      export LANG=C
> >      export LANGUAGE=C
> >      export LC_ALL=C
> > +
> > +    if [ -d ${TMPDIR}/aptkeys ]; then
> > +        for keyfile in ${TMPDIR}/aptkeys/*
> > +        do
> > +            kfn="$(basename $keyfile)"
> > +            cp $keyfile "$ROOTFSDIR/tmp/$kfn"
> > +            sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > "/tmp/$kfn"
> > +            rm "$ROOTFSDIR/tmp/$kfn"
> > +        done
> > +    fi
> >      sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> >          -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> >          -o Dir::Etc::sourceparts="-" \
> > @@ -128,6 +139,7 @@ setup_root_file_system() {
> >          sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture
> > ${DISTRO_ARCH} sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
> >      fi
> > +    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> >      sudo -E chroot "$ROOTFSDIR" \
> >          /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> >              ${IMAGE_TRANSIENT_PACKAGES}
> > diff --git a/meta/classes/isar-image.bbclass
> > b/meta/classes/isar-image.bbclass index cdd1651..4a89bd7 100644
> > --- a/meta/classes/isar-image.bbclass
> > +++ b/meta/classes/isar-image.bbclass
> > @@ -82,6 +82,7 @@ isar_image_cleanup() {
> >          fi
> >          rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
> >      '
> > +    sudo rm -f "${ISARKEYRING}"
> >  }
> >  
> >  do_rootfs() {
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index
> > a793585..b70d2a8 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -11,8
> > +11,8 @@ WORKDIR =
> > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
> > DEPLOY_ISAR_BOOTSTRAP =
> > "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
> > ISAR_BOOTSTRAP_LOCK =
> > "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock" -require
> > isar-bootstrap.inc inherit isar-bootstrap-helper +require
> > isar-bootstrap.inc do_generate_keyring[stamp-extra-info] =
> > "${DISTRO}-${DISTRO_ARCH}" 
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > 98c5c6b..8002a53 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,10 +23,8
> > @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> > -DEBOOTSTRAP_KEYRING = ""
> >  DEPLOY_ISAR_BOOTSTRAP ?= ""
> > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2"
> 
> When i talked about the OVERRIDES i meant also adding gnupg2 depending
> on a condition. And that condition would probably be, if
> DISTRO_APT_KEYS is non-empty.
> 
Pardon, my fault...


> DISTRO_APT_KEYS != "" && OVERRIDES += "gnupg-support"
> DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg-support = " gnupg2"
>  
> >  DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > file:///${REPO_BASE_DIR} \n" if
> > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> > @@ -43,7 +41,6 @@ python () { if own_pub_key:
> >              aptkeys += own_pub_key.split()
> >  
> > -    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> >      for key in aptkeys:
> >          d.appendVar("SRC_URI", " %s" % key)
> >          fetcher = bb.fetch2.Fetch([key], d)
> > @@ -171,13 +168,17 @@ def get_distro_components_argument(d, is_host):
> >      else:
> >          return ""
> >  
> > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > +
> > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> >  do_generate_keyring[dirs] = "${DL_DIR}"
> >  do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> >  do_generate_keyring() {
> >      if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > +        chmod 777 "${APTKEYTMPDIR}"
> >          for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > -           gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > -               --no-tty --homedir "${DL_DIR}"  --import "$keyfile"
> > +           cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> > +           sudo apt-key --keyring "${ISARKEYRING}" add "$keyfile"
> 
> This whole switching from gpg to "sudo chroot apt-key" would make a
> nice stand-alone patch, but i do not feel strong about that.
> 
I argue against that because I have a use case where I need all of this.
And I don't like small patches wandering around the mailing lists
without being merged.

Andreas
> Henning
> 
> >          done
> >      fi
> >  }
> > @@ -219,7 +220,6 @@ isar_bootstrap() {
> >              if [ ${IS_HOST} ]; then
> >                  ${DEBOOTSTRAP} $debootstrap_args \
> >                                 ${@get_distro_components_argument(d,
> > True)} \
> > -                               ${DEBOOTSTRAP_KEYRING} \
> >                                 "${@get_distro_suite(d, True)}" \
> >                                 "${ROOTFSDIR}" \
> >                                 "${@get_distro_source(d, True)}"
> > @@ -228,7 +228,6 @@ isar_bootstrap() {
> >                   "${DEBOOTSTRAP}" $debootstrap_args \
> >                                    --arch="${DISTRO_ARCH}" \
> >                                    ${@get_distro_components_argument(d,
> > False)} \
> > -                                  ${DEBOOTSTRAP_KEYRING} \
> >                                    "${@get_distro_suite(d, False)}" \
> >                                    "${ROOTFSDIR}" \
> >                                    "${@get_distro_source(d, False)}"
> 

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v3 6/6] If we use a custom keyring debootstrap may fall to https

[Andreas Reichel]

On Wed, Mar 06, 2019 at 06:43:11PM +0100, Henning Schild wrote:
> Am Wed, 6 Mar 2019 17:26:18 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> 
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> > 
> > So if we have something in aptkeyring, append https-support to
> > OVERRIDES.
> > 
> > Furthermore, the conditional append for https-support was missing
> > in qemuamd64-stretch.conf, thus, remove this from all the distros
> > and put it into the isar-bootstrap.inc.
> > 
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> >  meta-isar/conf/multiconfig/qemuamd64-buster.conf    |  1 -
> >  meta-isar/conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 11 +++++++++++
> >  3 files changed, 11 insertions(+), 2 deletions(-)
> > 
> > diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf
> > b/meta-isar/conf/multiconfig/qemuamd64-buster.conf index
> > 63df75c..da90993 100644 ---
> > a/meta-isar/conf/multiconfig/qemuamd64-buster.conf +++
> > b/meta-isar/conf/multiconfig/qemuamd64-buster.conf @@ -18,4 +18,3 @@
> > QEMU_MACHINE ?= "q35" QEMU_CPU ?= ""
> >  QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##
> > -bios /usr/local/share/ovmf/OVMF.fd" 
> > -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > apt-transport-https ca-certificates" diff --git
> > a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
> > b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf index
> > d1335ff..42c71df 100644 ---
> > a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf +++
> > b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf @@ -15,4 +15,3 @@
> > QEMU_MACHINE ?= "pc" QEMU_CPU ?= "" QEMU_DISK_ARGS ?= "-hda
> > ##ROOTFS_IMAGE##" 
> > -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > apt-transport-https ca-certificates" diff --git
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > 8002a53..64cefc6 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -25,6 +25,7
> > @@ BASEAPTSRCS = "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales
> > gnupg2" +DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > apt-transport-https ca-certificates" 
> >  DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > file:///${REPO_BASE_DIR} \n" if
> > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> > @@ -41,6 +42,12 @@ python () { if own_pub_key:
> >              aptkeys += own_pub_key.split()
> >  
> > +    if len(aptkeys) > 0:
> > +        # debootstrap falls back to https if there is no
> > +        # 'reliable' keyring, whatever that means, but it happened
> > +        # https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> > +        d.setVar("HAVE_CUSTOM_APT_KEYS", "True")
> 
> Why this indirection and the new variable? Can you not just check
> len(aptkeys) where you check HAVE_CUSTOM_APT_KEYS?
Because aptkeys is a local variable. And we don't want that to be
global.

> 
> In fact you probably want to look at DISTRO_APT_KEYS, which seems to be
> the source feeding all the other variables. Whatever you come up with
> in v3 4/6.
Not quite, becase we have DISTRO_APT_KEYS as well as BASE_REPO_KEY.
Both can contain several keys and feed aptkeys. (That's why I introduced
aptkeys, to remove the duplicate code in the keyring generation.)
> 
> >      for key in aptkeys:
> >          d.appendVar("SRC_URI", " %s" % key)
> >          fetcher = bb.fetch2.Fetch([key], d)
> > @@ -150,6 +157,10 @@ def get_distro_have_https_source(d,
> > is_host=False): return any(source[2].startswith("https://") for
> > source in generate_distro_sources(d, is_host)) 
> >  def get_distro_needs_https_support(d, is_host=False):
> > +    apt_keys = d.getVar("HAVE_CUSTOM_APT_KEYS", False)
> > +    if apt_keys:
> > +        return "https-support"
> > +
> 
> Reusing the OVERRIDE seems like a good idea to avoid double adding the
> packages. So gnupg implies https but does not add the packages again.
> But i think i would do something like
> 
> get_gnupg_overrides() 
>  ret = "gnupg"
>  # blabla bug
>  ret += " https-support"
> 
Sure. That makes sense, but I would like to keep the https-support part
in the other function, because that is there to tell all cases where
the image needs https-support, no matter wether apt-keys or https sources...

Gnupg then should go into a new function analog to the other one like
'get_distro_needs_gnupg_support' to enable a new gnupg OVERRIDE>

Andreas
> Henning
> 
> >      if get_distro_have_https_source(d, is_host):
> >          return "https-support"
> >      else:
> 

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v3 1/6] Revert "isar-bootstrap: Allow to set local keys in DISTRO_APT_KEYS"

[Andreas Reichel]

On Wed, Mar 06, 2019 at 06:18:12PM +0100, Henning Schild wrote:
> I do not see the point of the revert. You are just going back to the way
> it was broken before. It also does not seem to document what went wrong.
> 
> Instead of that revert 2/6 needs a more verbose commit message maybe
> mentioning the previous patches that got it wrong.
> 
Okay I will explain in more detail that it breaks the path... with
examples :)

> Henning
> 
> Am Wed, 6 Mar 2019 17:26:13 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> 
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > This reverts commit af983a13b6f4cee5d4af5e5cf6318231e02775c9.
> > 
> > We do not want to guess where the downloaded file will be. This does
> > not work anymore if the key is downloaded from remote with a URL.
> > 
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
> >  1 file changed, 1 insertion(+), 1 deletion(-)
> > 
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > 234d339..eb9a9df 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -40,7 +40,7
> > @@ python () { d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > ${APTKEYRING}") for key in distro_apt_keys.split():
> >              url = urlparse(key)
> > -            filename = ''.join([wd, url.path])
> > +            filename = os.path.basename(url.path)
> >              d.appendVar("SRC_URI", " %s" % key)
> >              d.appendVar("APTKEYFILES", " %s" % filename)
> >      if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> 

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v3 5/6] Use all source lists in target root apt

[Claudius Heine]

Hi,

On Wed, 2019-03-06 at 18:27 +0100, [ext] Henning Schild wrote:
> Am Wed, 6 Mar 2019 17:26:17 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> 
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > When we only use isar-apt.list, we cannot add additional
> > repositories
> > since they are listed in the bootstrap list only.
> 
> As far as i understood some entries in the list are not "enabled" or
> fully working before we added the key. The reasoning behind this
> change
> is somehow gpg ... and that needs to go in here. Because that is the
> first violation of "we only update isar-apt". But the violation is so
> early in the bootstrap that it is probably ok.

I don't think so. That apt-get update is part of the isar-bootstrap-
helper, so it will be called late when the image is build. That could
cause conflicting versions between the buildchroot and the resulting
image.

The reason why the general apt-get update is called only in the isar-
bootstrap recipe is so that there are no index conflicts between all
images that base itself on the same isar-bootstrap rootfs.

Claudius

> 
> Henning
> 
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> >  meta/classes/isar-bootstrap-helper.bbclass | 5 +----
> >  1 file changed, 1 insertion(+), 4 deletions(-)
> > 
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index 1f24037..f9e3fba
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -130,10 +130,7 @@ setup_root_file_system() {
> >              rm "$ROOTFSDIR/tmp/$kfn"
> >          done
> >      fi
> > -    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > -        -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > -        -o Dir::Etc::sourceparts="-" \
> > -        -o APT::Get::List-Cleanup="0"
> > +    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
> >      # Add multiarch for cross-target
> >      if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then
> >          sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-
> > architecture
> > ${DISTRO_ARCH}

Re: [PATCH v3 6/6] If we use a custom keyring debootstrap may fall to https

[Henning Schild]

Am Thu, 7 Mar 2019 10:14:39 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:

> On Wed, Mar 06, 2019 at 06:43:11PM +0100, Henning Schild wrote:
> > Am Wed, 6 Mar 2019 17:26:18 +0100
> > schrieb "[ext] Andreas J. Reichel"
> > <andreas.reichel.ext@siemens.com>: 
> > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > 
> > > See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> > > 
> > > So if we have something in aptkeyring, append https-support to
> > > OVERRIDES.
> > > 
> > > Furthermore, the conditional append for https-support was missing
> > > in qemuamd64-stretch.conf, thus, remove this from all the distros
> > > and put it into the isar-bootstrap.inc.
> > > 
> > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > ---
> > >  meta-isar/conf/multiconfig/qemuamd64-buster.conf    |  1 -
> > >  meta-isar/conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 11
> > > +++++++++++ 3 files changed, 11 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf
> > > b/meta-isar/conf/multiconfig/qemuamd64-buster.conf index
> > > 63df75c..da90993 100644 ---
> > > a/meta-isar/conf/multiconfig/qemuamd64-buster.conf +++
> > > b/meta-isar/conf/multiconfig/qemuamd64-buster.conf @@ -18,4 +18,3
> > > @@ QEMU_MACHINE ?= "q35" QEMU_CPU ?= ""
> > >  QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##
> > > -bios /usr/local/share/ovmf/OVMF.fd" 
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > > apt-transport-https ca-certificates" diff --git
> > > a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
> > > b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf index
> > > d1335ff..42c71df 100644 ---
> > > a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf +++
> > > b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf @@ -15,4 +15,3
> > > @@ QEMU_MACHINE ?= "pc" QEMU_CPU ?= "" QEMU_DISK_ARGS ?= "-hda
> > > ##ROOTFS_IMAGE##" 
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > > apt-transport-https ca-certificates" diff --git
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > 8002a53..64cefc6 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -25,6
> > > +25,7 @@ BASEAPTSRCS = "${WORKDIR}/base-apt-sources" APTKEYFILES
> > > = "" DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES =
> > > "locales gnupg2"
> > > +DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > > apt-transport-https ca-certificates" DISTRO_APT_PREMIRRORS ?=
> > > "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n"
> > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO'))
> > > else "" }" @@ -41,6 +42,12 @@ python () { if own_pub_key:
> > >              aptkeys += own_pub_key.split()
> > >  
> > > +    if len(aptkeys) > 0:
> > > +        # debootstrap falls back to https if there is no
> > > +        # 'reliable' keyring, whatever that means, but it
> > > happened
> > > +        #
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> > > +        d.setVar("HAVE_CUSTOM_APT_KEYS", "True")  
> > 
> > Why this indirection and the new variable? Can you not just check
> > len(aptkeys) where you check HAVE_CUSTOM_APT_KEYS?  
> Because aptkeys is a local variable. And we don't want that to be
> global.
> 
> > 
> > In fact you probably want to look at DISTRO_APT_KEYS, which seems
> > to be the source feeding all the other variables. Whatever you come
> > up with in v3 4/6.  
> Not quite, becase we have DISTRO_APT_KEYS as well as BASE_REPO_KEY.
> Both can contain several keys and feed aptkeys. (That's why I
> introduced aptkeys, to remove the duplicate code in the keyring
> generation.)
> >   
> > >      for key in aptkeys:
> > >          d.appendVar("SRC_URI", " %s" % key)
> > >          fetcher = bb.fetch2.Fetch([key], d)
> > > @@ -150,6 +157,10 @@ def get_distro_have_https_source(d,
> > > is_host=False): return any(source[2].startswith("https://") for
> > > source in generate_distro_sources(d, is_host)) 
> > >  def get_distro_needs_https_support(d, is_host=False):
> > > +    apt_keys = d.getVar("HAVE_CUSTOM_APT_KEYS", False)
> > > +    if apt_keys:
> > > +        return "https-support"
> > > +  
> > 
> > Reusing the OVERRIDE seems like a good idea to avoid double adding
> > the packages. So gnupg implies https but does not add the packages
> > again. But i think i would do something like
> > 
> > get_gnupg_overrides() 
> >  ret = "gnupg"
> >  # blabla bug
> >  ret += " https-support"
> >   
> Sure. That makes sense, but I would like to keep the https-support
> part in the other function, because that is there to tell all cases
> where the image needs https-support, no matter wether apt-keys or
> https sources...
> 
> Gnupg then should go into a new function analog to the other one like
> 'get_distro_needs_gnupg_support' to enable a new gnupg OVERRIDE>

Maybe one get_distro_overrides that just deals with the two cases we
currently have? Maybe keeping in mind that i.e the packages that
wic-img needs should maybe be an override as well.

Henning

> Andreas
> > Henning
> >   
> > >      if get_distro_have_https_source(d, is_host):
> > >          return "https-support"
> > >      else:  
> >   
> 

Re: [PATCH v3 4/6] Use apt-key to generate apt-keyring

[Henning Schild]

Am Thu, 7 Mar 2019 10:06:17 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:

> On Wed, Mar 06, 2019 at 06:33:12PM +0100, Henning Schild wrote:
> > Am Wed, 6 Mar 2019 17:26:16 +0100
> > schrieb "[ext] Andreas J. Reichel"
> > <andreas.reichel.ext@siemens.com>: 
> > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > 
> > > Use apt-key instead of manually calling gpg.
> > > 
> > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > ---
> > >  meta/classes/isar-bootstrap-helper.bbclass        | 12
> > > ++++++++++++ meta/classes/isar-image.bbclass                   |
> > > 1 + .../isar-bootstrap/isar-bootstrap-host.bb         |  2 +-
> > >  .../isar-bootstrap/isar-bootstrap.inc             | 15
> > > +++++++-------- 4 files changed, 21 insertions(+), 9 deletions(-)
> > > 
> > > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > > b/meta/classes/isar-bootstrap-helper.bbclass index
> > > d780b85..1f24037 100644 ---
> > > a/meta/classes/isar-bootstrap-helper.bbclass +++
> > > b/meta/classes/isar-bootstrap-helper.bbclass @@ -22,6 +22,7 @@
> > > HOST_DISTRO ?= "debian-stretch" HOST_ARCH ?=
> > > "${@get_deb_host_arch()}" 
> > >  HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
> > > +ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
> > >  
> > >  def reverse_bb_array(d, varname):
> > >      array = d.getVar(varname, True)
> > > @@ -119,6 +120,16 @@ setup_root_file_system() {
> > >      export LANG=C
> > >      export LANGUAGE=C
> > >      export LC_ALL=C
> > > +
> > > +    if [ -d ${TMPDIR}/aptkeys ]; then
> > > +        for keyfile in ${TMPDIR}/aptkeys/*
> > > +        do
> > > +            kfn="$(basename $keyfile)"
> > > +            cp $keyfile "$ROOTFSDIR/tmp/$kfn"
> > > +            sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > > "/tmp/$kfn"
> > > +            rm "$ROOTFSDIR/tmp/$kfn"
> > > +        done
> > > +    fi
> > >      sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > >          -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > >          -o Dir::Etc::sourceparts="-" \
> > > @@ -128,6 +139,7 @@ setup_root_file_system() {
> > >          sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg
> > > --add-architecture ${DISTRO_ARCH} sudo -E chroot
> > > "$ROOTFSDIR" /usr/bin/apt-get update fi
> > > +    sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> > >      sudo -E chroot "$ROOTFSDIR" \
> > >          /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> > >              ${IMAGE_TRANSIENT_PACKAGES}
> > > diff --git a/meta/classes/isar-image.bbclass
> > > b/meta/classes/isar-image.bbclass index cdd1651..4a89bd7 100644
> > > --- a/meta/classes/isar-image.bbclass
> > > +++ b/meta/classes/isar-image.bbclass
> > > @@ -82,6 +82,7 @@ isar_image_cleanup() {
> > >          fi
> > >          rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
> > >      '
> > > +    sudo rm -f "${ISARKEYRING}"
> > >  }
> > >  
> > >  do_rootfs() {
> > > diff --git
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index
> > > a793585..b70d2a8 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@
> > > -11,8 +11,8 @@ WORKDIR =
> > > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
> > > DEPLOY_ISAR_BOOTSTRAP =
> > > "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
> > > ISAR_BOOTSTRAP_LOCK =
> > > "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
> > > -require isar-bootstrap.inc inherit isar-bootstrap-helper
> > > +require isar-bootstrap.inc do_generate_keyring[stamp-extra-info]
> > > = "${DISTRO}-${DISTRO_ARCH}" diff --git
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > 98c5c6b..8002a53 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,10
> > > +23,8 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > > "${WORKDIR}/base-apt-sources" APTKEYFILES = "" -APTKEYRING =
> > > "${WORKDIR}/apt-keyring.gpg" -DEBOOTSTRAP_KEYRING = ""
> > >  DEPLOY_ISAR_BOOTSTRAP ?= ""
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2"  
> > 
> > When i talked about the OVERRIDES i meant also adding gnupg2
> > depending on a condition. And that condition would probably be, if
> > DISTRO_APT_KEYS is non-empty.
> >   
> Pardon, my fault...
> 
> 
> > DISTRO_APT_KEYS != "" && OVERRIDES += "gnupg-support"
> > DISTRO_BOOTSTRAP_BASE_PACKAGES_append_gnupg-support = " gnupg2"
> >    
> > >  DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > > file:///${REPO_BASE_DIR} \n" if
> > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else
> > > "" }" @@ -43,7 +41,6 @@ python () { if own_pub_key:
> > >              aptkeys += own_pub_key.split()
> > >  
> > > -    d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> > >      for key in aptkeys:
> > >          d.appendVar("SRC_URI", " %s" % key)
> > >          fetcher = bb.fetch2.Fetch([key], d)
> > > @@ -171,13 +168,17 @@ def get_distro_components_argument(d,
> > > is_host): else:
> > >          return ""
> > >  
> > > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > > +
> > > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> > >  do_generate_keyring[dirs] = "${DL_DIR}"
> > >  do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> > >  do_generate_keyring() {
> > >      if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > > +        chmod 777 "${APTKEYTMPDIR}"
> > >          for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > > -           gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > > -               --no-tty --homedir "${DL_DIR}"  --import
> > > "$keyfile"
> > > +           cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename
> > > "$keyfile")"
> > > +           sudo apt-key --keyring "${ISARKEYRING}" add
> > > "$keyfile"  
> > 
> > This whole switching from gpg to "sudo chroot apt-key" would make a
> > nice stand-alone patch, but i do not feel strong about that.
> >   
> I argue against that because I have a use case where I need all of
> this. And I don't like small patches wandering around the mailing
> lists without being merged.

Well one in the same queue/series of cause, which would express the
order/need while being small and focused.

Henning

> Andreas
> > Henning
> >   
> > >          done
> > >      fi
> > >  }
> > > @@ -219,7 +220,6 @@ isar_bootstrap() {
> > >              if [ ${IS_HOST} ]; then
> > >                  ${DEBOOTSTRAP} $debootstrap_args \
> > >                                 ${@get_distro_components_argument(d,
> > > True)} \
> > > -                               ${DEBOOTSTRAP_KEYRING} \
> > >                                 "${@get_distro_suite(d, True)}" \
> > >                                 "${ROOTFSDIR}" \
> > >                                 "${@get_distro_source(d, True)}"
> > > @@ -228,7 +228,6 @@ isar_bootstrap() {
> > >                   "${DEBOOTSTRAP}" $debootstrap_args \
> > >                                    --arch="${DISTRO_ARCH}" \
> > >                                    ${@get_distro_components_argument(d,
> > > False)} \
> > > -                                  ${DEBOOTSTRAP_KEYRING} \
> > >                                    "${@get_distro_suite(d,
> > > False)}" \ "${ROOTFSDIR}" \
> > >                                    "${@get_distro_source(d,
> > > False)}"  
> >   
> 

Re: [PATCH v3 6/6] If we use a custom keyring debootstrap may fall to https

[Henning Schild]

Am Thu, 7 Mar 2019 10:14:39 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:

> On Wed, Mar 06, 2019 at 06:43:11PM +0100, Henning Schild wrote:
> > Am Wed, 6 Mar 2019 17:26:18 +0100
> > schrieb "[ext] Andreas J. Reichel"
> > <andreas.reichel.ext@siemens.com>: 
> > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > 
> > > See https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> > > 
> > > So if we have something in aptkeyring, append https-support to
> > > OVERRIDES.
> > > 
> > > Furthermore, the conditional append for https-support was missing
> > > in qemuamd64-stretch.conf, thus, remove this from all the distros
> > > and put it into the isar-bootstrap.inc.
> > > 
> > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > ---
> > >  meta-isar/conf/multiconfig/qemuamd64-buster.conf    |  1 -
> > >  meta-isar/conf/multiconfig/qemuamd64-jessie.conf    |  1 -
> > >  meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 11
> > > +++++++++++ 3 files changed, 11 insertions(+), 2 deletions(-)
> > > 
> > > diff --git a/meta-isar/conf/multiconfig/qemuamd64-buster.conf
> > > b/meta-isar/conf/multiconfig/qemuamd64-buster.conf index
> > > 63df75c..da90993 100644 ---
> > > a/meta-isar/conf/multiconfig/qemuamd64-buster.conf +++
> > > b/meta-isar/conf/multiconfig/qemuamd64-buster.conf @@ -18,4 +18,3
> > > @@ QEMU_MACHINE ?= "q35" QEMU_CPU ?= ""
> > >  QEMU_DISK_ARGS ?= "-hda ##ROOTFS_IMAGE##
> > > -bios /usr/local/share/ovmf/OVMF.fd" 
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > > apt-transport-https ca-certificates" diff --git
> > > a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf
> > > b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf index
> > > d1335ff..42c71df 100644 ---
> > > a/meta-isar/conf/multiconfig/qemuamd64-jessie.conf +++
> > > b/meta-isar/conf/multiconfig/qemuamd64-jessie.conf @@ -15,4 +15,3
> > > @@ QEMU_MACHINE ?= "pc" QEMU_CPU ?= "" QEMU_DISK_ARGS ?= "-hda
> > > ##ROOTFS_IMAGE##" 
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > > apt-transport-https ca-certificates" diff --git
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > 8002a53..64cefc6 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -25,6
> > > +25,7 @@ BASEAPTSRCS = "${WORKDIR}/base-apt-sources" APTKEYFILES
> > > = "" DEPLOY_ISAR_BOOTSTRAP ?= "" DISTRO_BOOTSTRAP_BASE_PACKAGES =
> > > "locales gnupg2"
> > > +DISTRO_BOOTSTRAP_BASE_PACKAGES_append_https-support = "
> > > apt-transport-https ca-certificates" DISTRO_APT_PREMIRRORS ?=
> > > "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n"
> > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO'))
> > > else "" }" @@ -41,6 +42,12 @@ python () { if own_pub_key:
> > >              aptkeys += own_pub_key.split()
> > >  
> > > +    if len(aptkeys) > 0:
> > > +        # debootstrap falls back to https if there is no
> > > +        # 'reliable' keyring, whatever that means, but it
> > > happened
> > > +        #
> > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=891009
> > > +        d.setVar("HAVE_CUSTOM_APT_KEYS", "True")  
> > 
> > Why this indirection and the new variable? Can you not just check
> > len(aptkeys) where you check HAVE_CUSTOM_APT_KEYS?  
> Because aptkeys is a local variable. And we don't want that to be
> global.
> 
> > 
> > In fact you probably want to look at DISTRO_APT_KEYS, which seems
> > to be the source feeding all the other variables. Whatever you come
> > up with in v3 4/6.  
> Not quite, becase we have DISTRO_APT_KEYS as well as BASE_REPO_KEY.
> Both can contain several keys and feed aptkeys. (That's why I
> introduced aptkeys, to remove the duplicate code in the keyring
> generation.)

That is not your fault. But DISTRO_APT_KEYS should simply always have
BASE_REPO_KEY appended by default, which gets you back to one variable
that lists all repo keys. Remotes and the cache.

Henning

> >   
> > >      for key in aptkeys:
> > >          d.appendVar("SRC_URI", " %s" % key)
> > >          fetcher = bb.fetch2.Fetch([key], d)
> > > @@ -150,6 +157,10 @@ def get_distro_have_https_source(d,
> > > is_host=False): return any(source[2].startswith("https://") for
> > > source in generate_distro_sources(d, is_host)) 
> > >  def get_distro_needs_https_support(d, is_host=False):
> > > +    apt_keys = d.getVar("HAVE_CUSTOM_APT_KEYS", False)
> > > +    if apt_keys:
> > > +        return "https-support"
> > > +  
> > 
> > Reusing the OVERRIDE seems like a good idea to avoid double adding
> > the packages. So gnupg implies https but does not add the packages
> > again. But i think i would do something like
> > 
> > get_gnupg_overrides() 
> >  ret = "gnupg"
> >  # blabla bug
> >  ret += " https-support"
> >   
> Sure. That makes sense, but I would like to keep the https-support
> part in the other function, because that is there to tell all cases
> where the image needs https-support, no matter wether apt-keys or
> https sources...
> 
> Gnupg then should go into a new function analog to the other one like
> 'get_distro_needs_gnupg_support' to enable a new gnupg OVERRIDE>
> 
> Andreas
> > Henning
> >   
> > >      if get_distro_have_https_source(d, is_host):
> > >          return "https-support"
> > >      else:  
> >   
>