* [PATCH v2 0/3] Fixes usage of additional apt keys
@ 2019-02-27 15:18 Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 1/3] Fix and simplify apt keyring generation Andreas J. Reichel
` (3 more replies)
0 siblings, 4 replies; 16+ messages in thread
From: Andreas J. Reichel @ 2019-02-27 15:18 UTC (permalink / raw)
To: isar-users; +Cc: Andreas Reichel
From: Andreas Reichel <andreas.reichel.ext@siemens.com>
If the user does not want to replace the bootstrap source together with
the key, additional keys did not work.
This is solved by not replacing the keyring but to add keys to
/ect/apt/trusted.gpg.d/isar.gpg, where debootstrap and any apt call
can find it.
Furthermore, the code to add keys is simplified a lot by not manually
parsing URIs and guessing about download locations as well as
not manually handling gpg and giving apt config overrides.
It is much simpler by using `apt-key` and default apt keyring paths.
Furthermore, apt-get must not use a given single source list which was
used from debootstrapping. Otherwise, additional packages are always
unauthenticated, which is a quite misleading error. Instead, apt-get
should use all source lists available in the built root.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
Andreas Reichel (3):
Fix and simplify apt keyring generation
Use all source lists in target root apt
Separate apt-key entries from default keyring
meta/classes/isar-bootstrap-helper.bbclass | 17 ++++++--
meta/classes/isar-image.bbclass | 1 +
.../isar-bootstrap/isar-bootstrap-host.bb | 2 +-
.../isar-bootstrap/isar-bootstrap.inc | 39 +++++++++----------
4 files changed, 33 insertions(+), 26 deletions(-)
--
2.21.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-02-27 15:18 [PATCH v2 0/3] Fixes usage of additional apt keys Andreas J. Reichel
@ 2019-02-27 15:18 ` Andreas J. Reichel
2019-02-27 16:12 ` Henning Schild
2019-02-27 15:18 ` [PATCH v2 2/3] Use all source lists in target root apt Andreas J. Reichel
` (2 subsequent siblings)
3 siblings, 1 reply; 16+ messages in thread
From: Andreas J. Reichel @ 2019-02-27 15:18 UTC (permalink / raw)
To: isar-users; +Cc: Andreas Reichel
From: Andreas Reichel <andreas.reichel.ext@siemens.com>
Different fetcher stored keys in different sub dirs, and we can never
be sure about where downloaded files go.
To avoid this without making any assumptions, we ask the fetcher where
the file will be after it is downloaded. This way we also don't need to
parse the URL manually.
The code is simplified by removing duplicate code and using apt-key
instead of manually calling gpg.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
.../isar-bootstrap/isar-bootstrap.inc | 39 +++++++++----------
2 files changed, 29 insertions(+), 21 deletions(-)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index d780b85..b8c41f9 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -119,6 +119,16 @@ setup_root_file_system() {
export LANG=C
export LANGUAGE=C
export LC_ALL=C
+
+ if [ -d ${TMPDIR}/aptkeys ]; then
+ for keyfile in ${TMPDIR}/aptkeys/*
+ do
+ kfn="tmp/$(basename $keyfile)"
+ cp $keyfile "$ROOTFSDIR/$kfn"
+ sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add "/$kfn"
+ rm "$ROOTFSDIR/$kfn"
+ done
+ fi
sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
-o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
-o Dir::Etc::sourceparts="-" \
@@ -128,6 +138,7 @@ setup_root_file_system() {
sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
fi
+ sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
sudo -E chroot "$ROOTFSDIR" \
/usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
${IMAGE_TRANSIENT_PACKAGES}
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 234d339..2ef3b1e 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -23,35 +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources"
APTSRCS_INIT = "${WORKDIR}/apt-sources-init"
BASEAPTSRCS = "${WORKDIR}/base-apt-sources"
APTKEYFILES = ""
-APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
-DEBOOTSTRAP_KEYRING = ""
DEPLOY_ISAR_BOOTSTRAP ?= ""
-DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
+DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2 apt-transport-https ca-certificates"
DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org file:///${REPO_BASE_DIR} \n" if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
inherit base-apt-helper
python () {
- from urllib.parse import urlparse
distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
- wd = d.getVar("WORKDIR", True)
+ aptkeys = []
+
if distro_apt_keys:
- d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
- for key in distro_apt_keys.split():
- url = urlparse(key)
- filename = ''.join([wd, url.path])
- d.appendVar("SRC_URI", " %s" % key)
- d.appendVar("APTKEYFILES", " %s" % filename)
+ aptkeys += distro_apt_keys.split()
+
if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
own_pub_key = d.getVar("BASE_REPO_KEY", False)
if own_pub_key:
- d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
- for key in own_pub_key.split():
- url = urlparse(key)
- filename = ''.join([wd, url.path])
- d.appendVar("SRC_URI", " %s" % key)
- d.appendVar("APTKEYFILES", " %s" % filename)
+ aptkeys += own_pub_keys.split()
+
+ for key in aptkeys:
+ d.appendVar("SRC_URI", " %s" % key)
+ fetcher = bb.fetch2.Fetch([key], d)
+ filename = fetcher.localpath(key)
+ d.appendVar("APTKEYFILES", " %s" % filename)
}
def aggregate_files(d, file_list, file_out):
@@ -174,13 +169,17 @@ def get_distro_components_argument(d, is_host):
else:
return ""
+APTKEYTMPDIR := "${TMPDIR}/aptkeys"
+
+do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
do_generate_keyring[dirs] = "${DL_DIR}"
do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
do_generate_keyring() {
if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
+ chmod 777 "${APTKEYTMPDIR}"
for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
- gpg --no-default-keyring --keyring "${APTKEYRING}" \
- --no-tty --homedir "${DL_DIR}" --import "$keyfile"
+ cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
+ sudo apt-key add "$keyfile"
done
fi
}
@@ -222,7 +221,6 @@ isar_bootstrap() {
if [ ${IS_HOST} ]; then
${DEBOOTSTRAP} $debootstrap_args \
${@get_distro_components_argument(d, True)} \
- ${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, True)}" \
"${ROOTFSDIR}" \
"${@get_distro_source(d, True)}"
@@ -231,7 +229,6 @@ isar_bootstrap() {
"${DEBOOTSTRAP}" $debootstrap_args \
--arch="${DISTRO_ARCH}" \
${@get_distro_components_argument(d, False)} \
- ${DEBOOTSTRAP_KEYRING} \
"${@get_distro_suite(d, False)}" \
"${ROOTFSDIR}" \
"${@get_distro_source(d, False)}"
--
2.21.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 2/3] Use all source lists in target root apt
2019-02-27 15:18 [PATCH v2 0/3] Fixes usage of additional apt keys Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 1/3] Fix and simplify apt keyring generation Andreas J. Reichel
@ 2019-02-27 15:18 ` Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 3/3] Separate apt-key entries from default keyring Andreas J. Reichel
2019-02-27 16:15 ` [PATCH v2 0/3] Fixes usage of additional apt keys Henning Schild
3 siblings, 0 replies; 16+ messages in thread
From: Andreas J. Reichel @ 2019-02-27 15:18 UTC (permalink / raw)
To: isar-users; +Cc: Andreas Reichel
From: Andreas Reichel <andreas.reichel.ext@siemens.com>
When we only use isar-apt.list, we cannot add additional repositories
since they are listed in the bootstrap list only.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
meta/classes/isar-bootstrap-helper.bbclass | 5 +----
1 file changed, 1 insertion(+), 4 deletions(-)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index b8c41f9..26abf62 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -129,10 +129,7 @@ setup_root_file_system() {
rm "$ROOTFSDIR/$kfn"
done
fi
- sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
- -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
- -o Dir::Etc::sourceparts="-" \
- -o APT::Get::List-Cleanup="0"
+ sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
# Add multiarch for cross-target
if [ "${ROOTFS_ARCH}" != "${DISTRO_ARCH}" ]; then
sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture ${DISTRO_ARCH}
--
2.21.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* [PATCH v2 3/3] Separate apt-key entries from default keyring
2019-02-27 15:18 [PATCH v2 0/3] Fixes usage of additional apt keys Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 1/3] Fix and simplify apt keyring generation Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 2/3] Use all source lists in target root apt Andreas J. Reichel
@ 2019-02-27 15:18 ` Andreas J. Reichel
2019-02-27 16:14 ` Henning Schild
2019-02-27 16:15 ` [PATCH v2 0/3] Fixes usage of additional apt keys Henning Schild
3 siblings, 1 reply; 16+ messages in thread
From: Andreas J. Reichel @ 2019-02-27 15:18 UTC (permalink / raw)
To: isar-users; +Cc: Andreas Reichel
From: Andreas Reichel <andreas.reichel.ext@siemens.com>
Per default, apt-key add adds keys to /etc/apt/trusted.gpg.
However, when building without a container, we don't want to contaminate
the host. Therefore, we specify a keyring file in /etc/apt/trusted.gpg.d
directory named `isar.gpg`. This file can be deleted after the build.
This is necessary because we don't want to specify single keyrings
to debootstrap since we might need a mixture of several keyrings
as per default and would have to find all needed keyrings on the system,
export their keys and reimport into the build keyring, which is more
complicated and unneeded this way.
Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
---
meta/classes/isar-bootstrap-helper.bbclass | 1 +
meta/classes/isar-image.bbclass | 1 +
meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb | 2 +-
meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
4 files changed, 4 insertions(+), 2 deletions(-)
diff --git a/meta/classes/isar-bootstrap-helper.bbclass b/meta/classes/isar-bootstrap-helper.bbclass
index 26abf62..769cbef 100644
--- a/meta/classes/isar-bootstrap-helper.bbclass
+++ b/meta/classes/isar-bootstrap-helper.bbclass
@@ -22,6 +22,7 @@ HOST_DISTRO ?= "debian-stretch"
HOST_ARCH ?= "${@get_deb_host_arch()}"
HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
+ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
def reverse_bb_array(d, varname):
array = d.getVar(varname, True)
diff --git a/meta/classes/isar-image.bbclass b/meta/classes/isar-image.bbclass
index cdd1651..4a89bd7 100644
--- a/meta/classes/isar-image.bbclass
+++ b/meta/classes/isar-image.bbclass
@@ -82,6 +82,7 @@ isar_image_cleanup() {
fi
rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
'
+ sudo rm -f "${ISARKEYRING}"
}
do_rootfs() {
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
index a793585..b70d2a8 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
@@ -11,8 +11,8 @@ WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
DEPLOY_ISAR_BOOTSTRAP = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
ISAR_BOOTSTRAP_LOCK = "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock"
-require isar-bootstrap.inc
inherit isar-bootstrap-helper
+require isar-bootstrap.inc
do_generate_keyring[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index 2ef3b1e..4613732 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -179,7 +179,7 @@ do_generate_keyring() {
chmod 777 "${APTKEYTMPDIR}"
for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
- sudo apt-key add "$keyfile"
+ sudo apt-key --keyring "${ISARKEYRING}" add "$keyfile"
done
fi
}
--
2.21.0
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-02-27 15:18 ` [PATCH v2 1/3] Fix and simplify apt keyring generation Andreas J. Reichel
@ 2019-02-27 16:12 ` Henning Schild
2019-03-01 9:09 ` Andreas Reichel
` (2 more replies)
0 siblings, 3 replies; 16+ messages in thread
From: Henning Schild @ 2019-02-27 16:12 UTC (permalink / raw)
To: [ext] Andreas J. Reichel; +Cc: isar-users
Am Wed, 27 Feb 2019 16:18:54 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>
> Different fetcher stored keys in different sub dirs, and we can never
> be sure about where downloaded files go.
> To avoid this without making any assumptions, we ask the fetcher where
> the file will be after it is downloaded. This way we also don't need
> to parse the URL manually.
>
> The code is simplified by removing duplicate code and using apt-key
> instead of manually calling gpg.
>
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
> .../isar-bootstrap/isar-bootstrap.inc | 39
> +++++++++---------- 2 files changed, 29 insertions(+), 21 deletions(-)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index d780b85..b8c41f9
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -119,6 +119,16 @@ setup_root_file_system() {
> export LANG=C
> export LANGUAGE=C
> export LC_ALL=C
> +
> + if [ -d ${TMPDIR}/aptkeys ]; then
> + for keyfile in ${TMPDIR}/aptkeys/*
> + do
> + kfn="tmp/$(basename $keyfile)"
> + cp $keyfile "$ROOTFSDIR/$kfn"
> + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add "/$kfn"
Would probably be a good idea to work in /tmp/ as well and in a subdir.
Try naming your key "root" or "usr" ...
> + rm "$ROOTFSDIR/$kfn"
> + done
> + fi
> sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> -o Dir::Etc::sourceparts="-" \
> @@ -128,6 +138,7 @@ setup_root_file_system() {
> sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture
> ${DISTRO_ARCH} sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
> fi
> + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> sudo -E chroot "$ROOTFSDIR" \
> /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> ${IMAGE_TRANSIENT_PACKAGES}
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> 234d339..2ef3b1e 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,35
> +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> -DEBOOTSTRAP_KEYRING = ""
> DEPLOY_ISAR_BOOTSTRAP ?= ""
> -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2 apt-transport-https
> ca-certificates"
That looks more like leftovers from an experiment. Are you sure that is
needed?
In general i think that patch should be splittet into two or
even more patches. Because it solves at least two problems.
Henning
> DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> file:///${REPO_BASE_DIR} \n" if
> bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> inherit base-apt-helper
> python () {
> - from urllib.parse import urlparse
> distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> - wd = d.getVar("WORKDIR", True)
> + aptkeys = []
> +
> if distro_apt_keys:
> - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> - for key in distro_apt_keys.split():
> - url = urlparse(key)
> - filename = ''.join([wd, url.path])
> - d.appendVar("SRC_URI", " %s" % key)
> - d.appendVar("APTKEYFILES", " %s" % filename)
> + aptkeys += distro_apt_keys.split()
> +
> if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> own_pub_key = d.getVar("BASE_REPO_KEY", False)
> if own_pub_key:
> - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> ${APTKEYRING}")
> - for key in own_pub_key.split():
> - url = urlparse(key)
> - filename = ''.join([wd, url.path])
> - d.appendVar("SRC_URI", " %s" % key)
> - d.appendVar("APTKEYFILES", " %s" % filename)
> + aptkeys += own_pub_keys.split()
> +
> + for key in aptkeys:
> + d.appendVar("SRC_URI", " %s" % key)
> + fetcher = bb.fetch2.Fetch([key], d)
> + filename = fetcher.localpath(key)
> + d.appendVar("APTKEYFILES", " %s" % filename)
> }
>
> def aggregate_files(d, file_list, file_out):
> @@ -174,13 +169,17 @@ def get_distro_components_argument(d, is_host):
> else:
> return ""
>
> +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> +
> +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> do_generate_keyring[dirs] = "${DL_DIR}"
> do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> do_generate_keyring() {
> if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> + chmod 777 "${APTKEYTMPDIR}"
> for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> - gpg --no-default-keyring --keyring "${APTKEYRING}" \
> - --no-tty --homedir "${DL_DIR}" --import "$keyfile"
> + cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> + sudo apt-key add "$keyfile"
> done
> fi
> }
> @@ -222,7 +221,6 @@ isar_bootstrap() {
> if [ ${IS_HOST} ]; then
> ${DEBOOTSTRAP} $debootstrap_args \
> ${@get_distro_components_argument(d,
> True)} \
> - ${DEBOOTSTRAP_KEYRING} \
> "${@get_distro_suite(d, True)}" \
> "${ROOTFSDIR}" \
> "${@get_distro_source(d, True)}"
> @@ -231,7 +229,6 @@ isar_bootstrap() {
> "${DEBOOTSTRAP}" $debootstrap_args \
> --arch="${DISTRO_ARCH}" \
> ${@get_distro_components_argument(d,
> False)} \
> - ${DEBOOTSTRAP_KEYRING} \
> "${@get_distro_suite(d, False)}" \
> "${ROOTFSDIR}" \
> "${@get_distro_source(d, False)}"
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 3/3] Separate apt-key entries from default keyring
2019-02-27 15:18 ` [PATCH v2 3/3] Separate apt-key entries from default keyring Andreas J. Reichel
@ 2019-02-27 16:14 ` Henning Schild
2019-03-01 9:08 ` Andreas Reichel
0 siblings, 1 reply; 16+ messages in thread
From: Henning Schild @ 2019-02-27 16:14 UTC (permalink / raw)
To: [ext] Andreas J. Reichel; +Cc: isar-users
Is this fixing an issue introduced in p1? I guess you should just
squash it ;).
Henning
Am Wed, 27 Feb 2019 16:18:56 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>
> Per default, apt-key add adds keys to /etc/apt/trusted.gpg.
> However, when building without a container, we don't want to
> contaminate the host. Therefore, we specify a keyring file
> in /etc/apt/trusted.gpg.d directory named `isar.gpg`. This file can
> be deleted after the build.
>
> This is necessary because we don't want to specify single keyrings
> to debootstrap since we might need a mixture of several keyrings
> as per default and would have to find all needed keyrings on the
> system, export their keys and reimport into the build keyring, which
> is more complicated and unneeded this way.
>
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> ---
> meta/classes/isar-bootstrap-helper.bbclass | 1 +
> meta/classes/isar-image.bbclass | 1 +
> meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb | 2 +-
> meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
> 4 files changed, 4 insertions(+), 2 deletions(-)
>
> diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> b/meta/classes/isar-bootstrap-helper.bbclass index 26abf62..769cbef
> 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> +++ b/meta/classes/isar-bootstrap-helper.bbclass
> @@ -22,6 +22,7 @@ HOST_DISTRO ?= "debian-stretch"
> HOST_ARCH ?= "${@get_deb_host_arch()}"
>
> HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
> +ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
>
> def reverse_bb_array(d, varname):
> array = d.getVar(varname, True)
> diff --git a/meta/classes/isar-image.bbclass
> b/meta/classes/isar-image.bbclass index cdd1651..4a89bd7 100644
> --- a/meta/classes/isar-image.bbclass
> +++ b/meta/classes/isar-image.bbclass
> @@ -82,6 +82,7 @@ isar_image_cleanup() {
> fi
> rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
> '
> + sudo rm -f "${ISARKEYRING}"
> }
>
> do_rootfs() {
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index
> a793585..b70d2a8 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -11,8
> +11,8 @@ WORKDIR =
> "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
> DEPLOY_ISAR_BOOTSTRAP =
> "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
> ISAR_BOOTSTRAP_LOCK =
> "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock" -require
> isar-bootstrap.inc inherit isar-bootstrap-helper +require
> isar-bootstrap.inc do_generate_keyring[stamp-extra-info] =
> "${DISTRO}-${DISTRO_ARCH}"
> diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> 2ef3b1e..4613732 100644 ---
> a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -179,7
> +179,7 @@ do_generate_keyring() { chmod 777 "${APTKEYTMPDIR}"
> for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> - sudo apt-key add "$keyfile"
> + sudo apt-key --keyring "${ISARKEYRING}" add "$keyfile"
> done
> fi
> }
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 0/3] Fixes usage of additional apt keys
2019-02-27 15:18 [PATCH v2 0/3] Fixes usage of additional apt keys Andreas J. Reichel
` (2 preceding siblings ...)
2019-02-27 15:18 ` [PATCH v2 3/3] Separate apt-key entries from default keyring Andreas J. Reichel
@ 2019-02-27 16:15 ` Henning Schild
2019-03-01 9:17 ` Andreas Reichel
3 siblings, 1 reply; 16+ messages in thread
From: Henning Schild @ 2019-02-27 16:15 UTC (permalink / raw)
To: [ext] Andreas J. Reichel; +Cc: isar-users
Could you send a list of changes next time, or reply to the reviews.
Otherwise there is no way to tell what you did in response to the
review, which comments you skipped etc.
Henning
Am Wed, 27 Feb 2019 16:18:53 +0100
schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
>
> If the user does not want to replace the bootstrap source together
> with the key, additional keys did not work.
>
> This is solved by not replacing the keyring but to add keys to
> /ect/apt/trusted.gpg.d/isar.gpg, where debootstrap and any apt call
> can find it.
>
> Furthermore, the code to add keys is simplified a lot by not manually
> parsing URIs and guessing about download locations as well as
> not manually handling gpg and giving apt config overrides.
>
> It is much simpler by using `apt-key` and default apt keyring paths.
>
> Furthermore, apt-get must not use a given single source list which was
> used from debootstrapping. Otherwise, additional packages are always
> unauthenticated, which is a quite misleading error. Instead, apt-get
> should use all source lists available in the built root.
>
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
>
> Andreas Reichel (3):
> Fix and simplify apt keyring generation
> Use all source lists in target root apt
> Separate apt-key entries from default keyring
>
> meta/classes/isar-bootstrap-helper.bbclass | 17 ++++++--
> meta/classes/isar-image.bbclass | 1 +
> .../isar-bootstrap/isar-bootstrap-host.bb | 2 +-
> .../isar-bootstrap/isar-bootstrap.inc | 39
> +++++++++---------- 4 files changed, 33 insertions(+), 26 deletions(-)
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 3/3] Separate apt-key entries from default keyring
2019-02-27 16:14 ` Henning Schild
@ 2019-03-01 9:08 ` Andreas Reichel
0 siblings, 0 replies; 16+ messages in thread
From: Andreas Reichel @ 2019-03-01 9:08 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On Wed, Feb 27, 2019 at 05:14:43PM +0100, Henning Schild wrote:
> Is this fixing an issue introduced in p1? I guess you should just
> squash it ;).
>
> Henning
This is a matter of taste... as discussed for those building in a
container, it is no issue, for others, who don't it MIGHT be,
if they want to keep their keyring exactly as it was.
>
>
> Am Wed, 27 Feb 2019 16:18:56 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
>
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >
> > Per default, apt-key add adds keys to /etc/apt/trusted.gpg.
> > However, when building without a container, we don't want to
> > contaminate the host. Therefore, we specify a keyring file
> > in /etc/apt/trusted.gpg.d directory named `isar.gpg`. This file can
> > be deleted after the build.
> >
> > This is necessary because we don't want to specify single keyrings
> > to debootstrap since we might need a mixture of several keyrings
> > as per default and would have to find all needed keyrings on the
> > system, export their keys and reimport into the build keyring, which
> > is more complicated and unneeded this way.
> >
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> > meta/classes/isar-bootstrap-helper.bbclass | 1 +
> > meta/classes/isar-image.bbclass | 1 +
> > meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb | 2 +-
> > meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 2 +-
> > 4 files changed, 4 insertions(+), 2 deletions(-)
> >
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index 26abf62..769cbef
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -22,6 +22,7 @@ HOST_DISTRO ?= "debian-stretch"
> > HOST_ARCH ?= "${@get_deb_host_arch()}"
> >
> > HOST_DISTRO_APT_SOURCES += "conf/distro/${HOST_DISTRO}.list"
> > +ISARKEYRING = "/etc/apt/trusted.gpg.d/isar.gpg"
> >
> > def reverse_bb_array(d, varname):
> > array = d.getVar(varname, True)
> > diff --git a/meta/classes/isar-image.bbclass
> > b/meta/classes/isar-image.bbclass index cdd1651..4a89bd7 100644
> > --- a/meta/classes/isar-image.bbclass
> > +++ b/meta/classes/isar-image.bbclass
> > @@ -82,6 +82,7 @@ isar_image_cleanup() {
> > fi
> > rm -f "${IMAGE_ROOTFS}/etc/apt/sources-list"
> > '
> > + sudo rm -f "${ISARKEYRING}"
> > }
> >
> > do_rootfs() {
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb index
> > a793585..b70d2a8 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb @@ -11,8
> > +11,8 @@ WORKDIR =
> > "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}-${HOST_DISTRO}-${HOST_A
> > DEPLOY_ISAR_BOOTSTRAP =
> > "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}"
> > ISAR_BOOTSTRAP_LOCK =
> > "${DEPLOY_DIR_BOOTSTRAP}/${HOST_DISTRO}-${HOST_ARCH}.lock" -require
> > isar-bootstrap.inc inherit isar-bootstrap-helper +require
> > isar-bootstrap.inc do_generate_keyring[stamp-extra-info] =
> > "${DISTRO}-${DISTRO_ARCH}"
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > 2ef3b1e..4613732 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -179,7
> > +179,7 @@ do_generate_keyring() { chmod 777 "${APTKEYTMPDIR}"
> > for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> > - sudo apt-key add "$keyfile"
> > + sudo apt-key --keyring "${ISARKEYRING}" add "$keyfile"
> > done
> > fi
> > }
>
--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant
Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-02-27 16:12 ` Henning Schild
@ 2019-03-01 9:09 ` Andreas Reichel
2019-03-01 10:18 ` Henning Schild
2019-03-01 9:16 ` Andreas Reichel
2019-03-01 10:22 ` Henning Schild
2 siblings, 1 reply; 16+ messages in thread
From: Andreas Reichel @ 2019-03-01 9:09 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On Wed, Feb 27, 2019 at 05:12:42PM +0100, Henning Schild wrote:
> Am Wed, 27 Feb 2019 16:18:54 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
>
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >
> > Different fetcher stored keys in different sub dirs, and we can never
> > be sure about where downloaded files go.
> > To avoid this without making any assumptions, we ask the fetcher where
> > the file will be after it is downloaded. This way we also don't need
> > to parse the URL manually.
> >
> > The code is simplified by removing duplicate code and using apt-key
> > instead of manually calling gpg.
> >
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> > meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
> > .../isar-bootstrap/isar-bootstrap.inc | 39
> > +++++++++---------- 2 files changed, 29 insertions(+), 21 deletions(-)
> >
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index d780b85..b8c41f9
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -119,6 +119,16 @@ setup_root_file_system() {
> > export LANG=C
> > export LANGUAGE=C
> > export LC_ALL=C
> > +
> > + if [ -d ${TMPDIR}/aptkeys ]; then
> > + for keyfile in ${TMPDIR}/aptkeys/*
> > + do
> > + kfn="tmp/$(basename $keyfile)"
> > + cp $keyfile "$ROOTFSDIR/$kfn"
> > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add "/$kfn"
>
> Would probably be a good idea to work in /tmp/ as well and in a subdir.
> Try naming your key "root" or "usr" ...
>
> > + rm "$ROOTFSDIR/$kfn"
> > + done
> > + fi
> > sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > -o Dir::Etc::sourceparts="-" \
> > @@ -128,6 +138,7 @@ setup_root_file_system() {
> > sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg --add-architecture
> > ${DISTRO_ARCH} sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update
> > fi
> > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> > sudo -E chroot "$ROOTFSDIR" \
> > /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> > ${IMAGE_TRANSIENT_PACKAGES}
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > 234d339..2ef3b1e 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,35
> > +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> > -DEBOOTSTRAP_KEYRING = ""
> > DEPLOY_ISAR_BOOTSTRAP ?= ""
> > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2 apt-transport-https
> > ca-certificates"
>
> That looks more like leftovers from an experiment. Are you sure that is
> needed?
This is needed because debootstrap can switch to https by itself if it
is unsure about the sources (apt-transport-https, ca-certificates),
gnupg2 we need for apt-key.
>
>
> In general i think that patch should be splittet into two or
> even more patches. Because it solves at least two problems.
>
> Henning
>
> > DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > file:///${REPO_BASE_DIR} \n" if
> > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else "" }"
> > inherit base-apt-helper
> > python () {
> > - from urllib.parse import urlparse
> > distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> > - wd = d.getVar("WORKDIR", True)
> > + aptkeys = []
> > +
> > if distro_apt_keys:
> > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> > - for key in distro_apt_keys.split():
> > - url = urlparse(key)
> > - filename = ''.join([wd, url.path])
> > - d.appendVar("SRC_URI", " %s" % key)
> > - d.appendVar("APTKEYFILES", " %s" % filename)
> > + aptkeys += distro_apt_keys.split()
> > +
> > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> > own_pub_key = d.getVar("BASE_REPO_KEY", False)
> > if own_pub_key:
> > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > ${APTKEYRING}")
> > - for key in own_pub_key.split():
> > - url = urlparse(key)
> > - filename = ''.join([wd, url.path])
> > - d.appendVar("SRC_URI", " %s" % key)
> > - d.appendVar("APTKEYFILES", " %s" % filename)
> > + aptkeys += own_pub_keys.split()
> > +
> > + for key in aptkeys:
> > + d.appendVar("SRC_URI", " %s" % key)
> > + fetcher = bb.fetch2.Fetch([key], d)
> > + filename = fetcher.localpath(key)
> > + d.appendVar("APTKEYFILES", " %s" % filename)
> > }
> >
> > def aggregate_files(d, file_list, file_out):
> > @@ -174,13 +169,17 @@ def get_distro_components_argument(d, is_host):
> > else:
> > return ""
> >
> > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > +
> > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> > do_generate_keyring[dirs] = "${DL_DIR}"
> > do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> > do_generate_keyring() {
> > if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > + chmod 777 "${APTKEYTMPDIR}"
> > for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > - gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > - --no-tty --homedir "${DL_DIR}" --import "$keyfile"
> > + cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> > + sudo apt-key add "$keyfile"
> > done
> > fi
> > }
> > @@ -222,7 +221,6 @@ isar_bootstrap() {
> > if [ ${IS_HOST} ]; then
> > ${DEBOOTSTRAP} $debootstrap_args \
> > ${@get_distro_components_argument(d,
> > True)} \
> > - ${DEBOOTSTRAP_KEYRING} \
> > "${@get_distro_suite(d, True)}" \
> > "${ROOTFSDIR}" \
> > "${@get_distro_source(d, True)}"
> > @@ -231,7 +229,6 @@ isar_bootstrap() {
> > "${DEBOOTSTRAP}" $debootstrap_args \
> > --arch="${DISTRO_ARCH}" \
> > ${@get_distro_components_argument(d,
> > False)} \
> > - ${DEBOOTSTRAP_KEYRING} \
> > "${@get_distro_suite(d, False)}" \
> > "${ROOTFSDIR}" \
> > "${@get_distro_source(d, False)}"
>
--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant
Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-02-27 16:12 ` Henning Schild
2019-03-01 9:09 ` Andreas Reichel
@ 2019-03-01 9:16 ` Andreas Reichel
2019-03-01 10:11 ` Henning Schild
2019-03-01 10:22 ` Henning Schild
2 siblings, 1 reply; 16+ messages in thread
From: Andreas Reichel @ 2019-03-01 9:16 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On Wed, Feb 27, 2019 at 05:12:42PM +0100, Henning Schild wrote:
> Am Wed, 27 Feb 2019 16:18:54 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
>
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >
> > +
> > + if [ -d ${TMPDIR}/aptkeys ]; then
> > + for keyfile in ${TMPDIR}/aptkeys/*
> > + do
> > + kfn="tmp/$(basename $keyfile)"
> > + cp $keyfile "$ROOTFSDIR/$kfn"
> > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add "/$kfn"
>
> Would probably be a good idea to work in /tmp/ as well and in a subdir.
> Try naming your key "root" or "usr" ...
>
I AM working in /tmp/, ... a subdir is not needed since we directly work
with the file names.
This is not " a key ", these are arbitrarily many keys and the name of
the key files don't change between the host, the build and target
chroots.
Andreas
--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant
Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 0/3] Fixes usage of additional apt keys
2019-02-27 16:15 ` [PATCH v2 0/3] Fixes usage of additional apt keys Henning Schild
@ 2019-03-01 9:17 ` Andreas Reichel
0 siblings, 0 replies; 16+ messages in thread
From: Andreas Reichel @ 2019-03-01 9:17 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On Wed, Feb 27, 2019 at 05:15:56PM +0100, Henning Schild wrote:
> Could you send a list of changes next time, or reply to the reviews.
>
> Otherwise there is no way to tell what you did in response to the
There is a simple way... read the cover letters. But sure, I will
do better next time :)
Andreas
> review, which comments you skipped etc.
>
> Henning
>
> Am Wed, 27 Feb 2019 16:18:53 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
>
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >
> > If the user does not want to replace the bootstrap source together
> > with the key, additional keys did not work.
> >
> > This is solved by not replacing the keyring but to add keys to
> > /ect/apt/trusted.gpg.d/isar.gpg, where debootstrap and any apt call
> > can find it.
> >
> > Furthermore, the code to add keys is simplified a lot by not manually
> > parsing URIs and guessing about download locations as well as
> > not manually handling gpg and giving apt config overrides.
> >
> > It is much simpler by using `apt-key` and default apt keyring paths.
> >
> > Furthermore, apt-get must not use a given single source list which was
> > used from debootstrapping. Otherwise, additional packages are always
> > unauthenticated, which is a quite misleading error. Instead, apt-get
> > should use all source lists available in the built root.
> >
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >
> > Andreas Reichel (3):
> > Fix and simplify apt keyring generation
> > Use all source lists in target root apt
> > Separate apt-key entries from default keyring
> >
> > meta/classes/isar-bootstrap-helper.bbclass | 17 ++++++--
> > meta/classes/isar-image.bbclass | 1 +
> > .../isar-bootstrap/isar-bootstrap-host.bb | 2 +-
> > .../isar-bootstrap/isar-bootstrap.inc | 39
> > +++++++++---------- 4 files changed, 33 insertions(+), 26 deletions(-)
> >
>
--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant
Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-03-01 9:16 ` Andreas Reichel
@ 2019-03-01 10:11 ` Henning Schild
0 siblings, 0 replies; 16+ messages in thread
From: Henning Schild @ 2019-03-01 10:11 UTC (permalink / raw)
To: Andreas Reichel; +Cc: isar-users
Am Fri, 1 Mar 2019 10:16:50 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:
> On Wed, Feb 27, 2019 at 05:12:42PM +0100, Henning Schild wrote:
> > Am Wed, 27 Feb 2019 16:18:54 +0100
> > schrieb "[ext] Andreas J. Reichel"
> > <andreas.reichel.ext@siemens.com>:
> > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > >
> > > +
> > > + if [ -d ${TMPDIR}/aptkeys ]; then
> > > + for keyfile in ${TMPDIR}/aptkeys/*
> > > + do
> > > + kfn="tmp/$(basename $keyfile)"
> > > + cp $keyfile "$ROOTFSDIR/$kfn"
> > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > > "/$kfn"
> >
> > Would probably be a good idea to work in /tmp/ as well and in a
> > subdir. Try naming your key "root" or "usr" ...
> >
> I AM working in /tmp/, ... a subdir is not needed since we directly
> work with the file names.
> This is not " a key ", these are arbitrarily many keys and the name of
> the key files don't change between the host, the build and target
> chroots.
I see the "tmp/" is in $kfn, might be more readable to have in the the
cp and apt-key.
Henning
> Andreas
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-03-01 9:09 ` Andreas Reichel
@ 2019-03-01 10:18 ` Henning Schild
2019-03-06 16:29 ` Andreas Reichel
0 siblings, 1 reply; 16+ messages in thread
From: Henning Schild @ 2019-03-01 10:18 UTC (permalink / raw)
To: Andreas Reichel; +Cc: isar-users
Am Fri, 1 Mar 2019 10:09:49 +0100
schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:
> On Wed, Feb 27, 2019 at 05:12:42PM +0100, Henning Schild wrote:
> > Am Wed, 27 Feb 2019 16:18:54 +0100
> > schrieb "[ext] Andreas J. Reichel"
> > <andreas.reichel.ext@siemens.com>:
> > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > >
> > > Different fetcher stored keys in different sub dirs, and we can
> > > never be sure about where downloaded files go.
> > > To avoid this without making any assumptions, we ask the fetcher
> > > where the file will be after it is downloaded. This way we also
> > > don't need to parse the URL manually.
> > >
> > > The code is simplified by removing duplicate code and using
> > > apt-key instead of manually calling gpg.
> > >
> > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > ---
> > > meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
> > > .../isar-bootstrap/isar-bootstrap.inc | 39
> > > +++++++++---------- 2 files changed, 29 insertions(+), 21
> > > deletions(-)
> > >
> > > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > > b/meta/classes/isar-bootstrap-helper.bbclass index
> > > d780b85..b8c41f9 100644 ---
> > > a/meta/classes/isar-bootstrap-helper.bbclass +++
> > > b/meta/classes/isar-bootstrap-helper.bbclass @@ -119,6 +119,16 @@
> > > setup_root_file_system() { export LANG=C
> > > export LANGUAGE=C
> > > export LC_ALL=C
> > > +
> > > + if [ -d ${TMPDIR}/aptkeys ]; then
> > > + for keyfile in ${TMPDIR}/aptkeys/*
> > > + do
> > > + kfn="tmp/$(basename $keyfile)"
> > > + cp $keyfile "$ROOTFSDIR/$kfn"
> > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > > "/$kfn"
> >
> > Would probably be a good idea to work in /tmp/ as well and in a
> > subdir. Try naming your key "root" or "usr" ...
> >
> > > + rm "$ROOTFSDIR/$kfn"
> > > + done
> > > + fi
> > > sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > > -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > > -o Dir::Etc::sourceparts="-" \
> > > @@ -128,6 +138,7 @@ setup_root_file_system() {
> > > sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg
> > > --add-architecture ${DISTRO_ARCH} sudo -E chroot
> > > "$ROOTFSDIR" /usr/bin/apt-get update fi
> > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> > > sudo -E chroot "$ROOTFSDIR" \
> > > /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> > > ${IMAGE_TRANSIENT_PACKAGES}
> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > 234d339..2ef3b1e 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,35
> > > +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > > "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > > -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> > > -DEBOOTSTRAP_KEYRING = ""
> > > DEPLOY_ISAR_BOOTSTRAP ?= ""
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2
> > > apt-transport-https ca-certificates"
> >
> > That looks more like leftovers from an experiment. Are you sure
> > that is needed?
>
> This is needed because debootstrap can switch to https by itself if it
> is unsure about the sources (apt-transport-https, ca-certificates),
> gnupg2 we need for apt-key.
Adding the packages should probably depend on whether DISTRO_APT_KEYS
is non-empty. And adding the ones that the https-detector adds should
be tested, not that dual mention makes debootstrap crap out.
I guess the OVERRIDES method from 9196b820636a9 should be used for gpg
support as well.
Henning
> >
> >
> > In general i think that patch should be splittet into two or
> > even more patches. Because it solves at least two problems.
> >
> > Henning
> >
> > > DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > > file:///${REPO_BASE_DIR} \n" if
> > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else
> > > "" }" inherit base-apt-helper
> > > python () {
> > > - from urllib.parse import urlparse
> > > distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> > > - wd = d.getVar("WORKDIR", True)
> > > + aptkeys = []
> > > +
> > > if distro_apt_keys:
> > > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > > ${APTKEYRING}")
> > > - for key in distro_apt_keys.split():
> > > - url = urlparse(key)
> > > - filename = ''.join([wd, url.path])
> > > - d.appendVar("SRC_URI", " %s" % key)
> > > - d.appendVar("APTKEYFILES", " %s" % filename)
> > > + aptkeys += distro_apt_keys.split()
> > > +
> > > if
> > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> > > own_pub_key = d.getVar("BASE_REPO_KEY", False) if own_pub_key:
> > > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > > ${APTKEYRING}")
> > > - for key in own_pub_key.split():
> > > - url = urlparse(key)
> > > - filename = ''.join([wd, url.path])
> > > - d.appendVar("SRC_URI", " %s" % key)
> > > - d.appendVar("APTKEYFILES", " %s" % filename)
> > > + aptkeys += own_pub_keys.split()
> > > +
> > > + for key in aptkeys:
> > > + d.appendVar("SRC_URI", " %s" % key)
> > > + fetcher = bb.fetch2.Fetch([key], d)
> > > + filename = fetcher.localpath(key)
> > > + d.appendVar("APTKEYFILES", " %s" % filename)
> > > }
> > >
> > > def aggregate_files(d, file_list, file_out):
> > > @@ -174,13 +169,17 @@ def get_distro_components_argument(d,
> > > is_host): else:
> > > return ""
> > >
> > > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > > +
> > > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> > > do_generate_keyring[dirs] = "${DL_DIR}"
> > > do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> > > do_generate_keyring() {
> > > if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > > + chmod 777 "${APTKEYTMPDIR}"
> > > for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > > - gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > > - --no-tty --homedir "${DL_DIR}" --import
> > > "$keyfile"
> > > + cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename
> > > "$keyfile")"
> > > + sudo apt-key add "$keyfile"
> > > done
> > > fi
> > > }
> > > @@ -222,7 +221,6 @@ isar_bootstrap() {
> > > if [ ${IS_HOST} ]; then
> > > ${DEBOOTSTRAP} $debootstrap_args \
> > > ${@get_distro_components_argument(d,
> > > True)} \
> > > - ${DEBOOTSTRAP_KEYRING} \
> > > "${@get_distro_suite(d, True)}" \
> > > "${ROOTFSDIR}" \
> > > "${@get_distro_source(d, True)}"
> > > @@ -231,7 +229,6 @@ isar_bootstrap() {
> > > "${DEBOOTSTRAP}" $debootstrap_args \
> > > --arch="${DISTRO_ARCH}" \
> > > ${@get_distro_components_argument(d,
> > > False)} \
> > > - ${DEBOOTSTRAP_KEYRING} \
> > > "${@get_distro_suite(d,
> > > False)}" \ "${ROOTFSDIR}" \
> > > "${@get_distro_source(d,
> > > False)}"
> >
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-02-27 16:12 ` Henning Schild
2019-03-01 9:09 ` Andreas Reichel
2019-03-01 9:16 ` Andreas Reichel
@ 2019-03-01 10:22 ` Henning Schild
2019-03-06 16:29 ` Andreas Reichel
2 siblings, 1 reply; 16+ messages in thread
From: Henning Schild @ 2019-03-01 10:22 UTC (permalink / raw)
To: [ext] Andreas J. Reichel; +Cc: isar-users
Am Wed, 27 Feb 2019 17:12:42 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> Am Wed, 27 Feb 2019 16:18:54 +0100
> schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
>
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> >
> > Different fetcher stored keys in different sub dirs, and we can
> > never be sure about where downloaded files go.
> > To avoid this without making any assumptions, we ask the fetcher
> > where the file will be after it is downloaded. This way we also
> > don't need to parse the URL manually.
> >
> > The code is simplified by removing duplicate code and using apt-key
> > instead of manually calling gpg.
> >
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > ---
> > meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
> > .../isar-bootstrap/isar-bootstrap.inc | 39
> > +++++++++---------- 2 files changed, 29 insertions(+), 21
> > deletions(-)
> >
> > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > b/meta/classes/isar-bootstrap-helper.bbclass index d780b85..b8c41f9
> > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > @@ -119,6 +119,16 @@ setup_root_file_system() {
> > export LANG=C
> > export LANGUAGE=C
> > export LC_ALL=C
> > +
> > + if [ -d ${TMPDIR}/aptkeys ]; then
> > + for keyfile in ${TMPDIR}/aptkeys/*
> > + do
> > + kfn="tmp/$(basename $keyfile)"
> > + cp $keyfile "$ROOTFSDIR/$kfn"
> > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > "/$kfn"
>
> Would probably be a good idea to work in /tmp/ as well and in a
> subdir. Try naming your key "root" or "usr" ...
>
> > + rm "$ROOTFSDIR/$kfn"
> > + done
> > + fi
> > sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > -o Dir::Etc::sourceparts="-" \
> > @@ -128,6 +138,7 @@ setup_root_file_system() {
> > sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg
> > --add-architecture ${DISTRO_ARCH} sudo -E chroot
> > "$ROOTFSDIR" /usr/bin/apt-get update fi
> > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> > sudo -E chroot "$ROOTFSDIR" \
> > /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> > ${IMAGE_TRANSIENT_PACKAGES}
> > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > 234d339..2ef3b1e 100644 ---
> > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,35
> > +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> > -DEBOOTSTRAP_KEYRING = ""
> > DEPLOY_ISAR_BOOTSTRAP ?= ""
> > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2
> > apt-transport-https ca-certificates"
>
> That looks more like leftovers from an experiment. Are you sure that
> is needed?
>
>
> In general i think that patch should be splittet into two or
> even more patches. Because it solves at least two problems.
One possible split would be:
- gpg -> apt-key
- filename.join -> fetcher.localpath
- adding BASE_PACKAGES depending on GPG needed
- ...
Henning
> Henning
>
> > DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > file:///${REPO_BASE_DIR} \n" if
> > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else
> > "" }" inherit base-apt-helper
> > python () {
> > - from urllib.parse import urlparse
> > distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> > - wd = d.getVar("WORKDIR", True)
> > + aptkeys = []
> > +
> > if distro_apt_keys:
> > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> > - for key in distro_apt_keys.split():
> > - url = urlparse(key)
> > - filename = ''.join([wd, url.path])
> > - d.appendVar("SRC_URI", " %s" % key)
> > - d.appendVar("APTKEYFILES", " %s" % filename)
> > + aptkeys += distro_apt_keys.split()
> > +
> > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> > own_pub_key = d.getVar("BASE_REPO_KEY", False)
> > if own_pub_key:
> > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > ${APTKEYRING}")
> > - for key in own_pub_key.split():
> > - url = urlparse(key)
> > - filename = ''.join([wd, url.path])
> > - d.appendVar("SRC_URI", " %s" % key)
> > - d.appendVar("APTKEYFILES", " %s" % filename)
> > + aptkeys += own_pub_keys.split()
> > +
> > + for key in aptkeys:
> > + d.appendVar("SRC_URI", " %s" % key)
> > + fetcher = bb.fetch2.Fetch([key], d)
> > + filename = fetcher.localpath(key)
> > + d.appendVar("APTKEYFILES", " %s" % filename)
> > }
> >
> > def aggregate_files(d, file_list, file_out):
> > @@ -174,13 +169,17 @@ def get_distro_components_argument(d,
> > is_host): else:
> > return ""
> >
> > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > +
> > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> > do_generate_keyring[dirs] = "${DL_DIR}"
> > do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> > do_generate_keyring() {
> > if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > + chmod 777 "${APTKEYTMPDIR}"
> > for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > - gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > - --no-tty --homedir "${DL_DIR}" --import "$keyfile"
> > + cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> > + sudo apt-key add "$keyfile"
> > done
> > fi
> > }
> > @@ -222,7 +221,6 @@ isar_bootstrap() {
> > if [ ${IS_HOST} ]; then
> > ${DEBOOTSTRAP} $debootstrap_args \
> > ${@get_distro_components_argument(d,
> > True)} \
> > - ${DEBOOTSTRAP_KEYRING} \
> > "${@get_distro_suite(d, True)}" \
> > "${ROOTFSDIR}" \
> > "${@get_distro_source(d, True)}"
> > @@ -231,7 +229,6 @@ isar_bootstrap() {
> > "${DEBOOTSTRAP}" $debootstrap_args \
> > --arch="${DISTRO_ARCH}" \
> > ${@get_distro_components_argument(d,
> > False)} \
> > - ${DEBOOTSTRAP_KEYRING} \
> > "${@get_distro_suite(d, False)}"
> > \ "${ROOTFSDIR}" \
> > "${@get_distro_source(d,
> > False)}"
>
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-03-01 10:22 ` Henning Schild
@ 2019-03-06 16:29 ` Andreas Reichel
0 siblings, 0 replies; 16+ messages in thread
From: Andreas Reichel @ 2019-03-06 16:29 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On Fri, Mar 01, 2019 at 11:22:24AM +0100, Henning Schild wrote:
> Am Wed, 27 Feb 2019 17:12:42 +0100
> schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
>
> > Am Wed, 27 Feb 2019 16:18:54 +0100
> > schrieb "[ext] Andreas J. Reichel" <andreas.reichel.ext@siemens.com>:
> >
> > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > >
> > > Different fetcher stored keys in different sub dirs, and we can
> > > never be sure about where downloaded files go.
> > > To avoid this without making any assumptions, we ask the fetcher
> > > where the file will be after it is downloaded. This way we also
> > > don't need to parse the URL manually.
> > >
> > > The code is simplified by removing duplicate code and using apt-key
> > > instead of manually calling gpg.
> > >
> > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > ---
> > > meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
> > > .../isar-bootstrap/isar-bootstrap.inc | 39
> > > +++++++++---------- 2 files changed, 29 insertions(+), 21
> > > deletions(-)
> > >
> > > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > > b/meta/classes/isar-bootstrap-helper.bbclass index d780b85..b8c41f9
> > > 100644 --- a/meta/classes/isar-bootstrap-helper.bbclass
> > > +++ b/meta/classes/isar-bootstrap-helper.bbclass
> > > @@ -119,6 +119,16 @@ setup_root_file_system() {
> > > export LANG=C
> > > export LANGUAGE=C
> > > export LC_ALL=C
> > > +
> > > + if [ -d ${TMPDIR}/aptkeys ]; then
> > > + for keyfile in ${TMPDIR}/aptkeys/*
> > > + do
> > > + kfn="tmp/$(basename $keyfile)"
> > > + cp $keyfile "$ROOTFSDIR/$kfn"
> > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > > "/$kfn"
> >
> > Would probably be a good idea to work in /tmp/ as well and in a
> > subdir. Try naming your key "root" or "usr" ...
> >
Already worked in tmp :) But factored it out of $kfn in patch v3.
> > > + rm "$ROOTFSDIR/$kfn"
> > > + done
> > > + fi
> > > sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > > -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > > -o Dir::Etc::sourceparts="-" \
> > > @@ -128,6 +138,7 @@ setup_root_file_system() {
> > > sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg
> > > --add-architecture ${DISTRO_ARCH} sudo -E chroot
> > > "$ROOTFSDIR" /usr/bin/apt-get update fi
> > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> > > sudo -E chroot "$ROOTFSDIR" \
> > > /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> > > ${IMAGE_TRANSIENT_PACKAGES}
> > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > 234d339..2ef3b1e 100644 ---
> > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,35
> > > +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > > "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > > -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> > > -DEBOOTSTRAP_KEYRING = ""
> > > DEPLOY_ISAR_BOOTSTRAP ?= ""
> > > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2
> > > apt-transport-https ca-certificates"
> >
> > That looks more like leftovers from an experiment. Are you sure that
> > is needed?
> >
Fixed in v3
> >
> > In general i think that patch should be splittet into two or
> > even more patches. Because it solves at least two problems.
>
> One possible split would be:
>
> - gpg -> apt-key
> - filename.join -> fetcher.localpath
> - adding BASE_PACKAGES depending on GPG needed
> - ...
>
Done in v3.
> Henning
>
> > Henning
> >
> > > DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > > file:///${REPO_BASE_DIR} \n" if
> > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else
> > > "" }" inherit base-apt-helper
> > > python () {
> > > - from urllib.parse import urlparse
> > > distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> > > - wd = d.getVar("WORKDIR", True)
> > > + aptkeys = []
> > > +
> > > if distro_apt_keys:
> > > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring ${APTKEYRING}")
> > > - for key in distro_apt_keys.split():
> > > - url = urlparse(key)
> > > - filename = ''.join([wd, url.path])
> > > - d.appendVar("SRC_URI", " %s" % key)
> > > - d.appendVar("APTKEYFILES", " %s" % filename)
> > > + aptkeys += distro_apt_keys.split()
> > > +
> > > if bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> > > own_pub_key = d.getVar("BASE_REPO_KEY", False)
> > > if own_pub_key:
> > > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > > ${APTKEYRING}")
> > > - for key in own_pub_key.split():
> > > - url = urlparse(key)
> > > - filename = ''.join([wd, url.path])
> > > - d.appendVar("SRC_URI", " %s" % key)
> > > - d.appendVar("APTKEYFILES", " %s" % filename)
> > > + aptkeys += own_pub_keys.split()
> > > +
> > > + for key in aptkeys:
> > > + d.appendVar("SRC_URI", " %s" % key)
> > > + fetcher = bb.fetch2.Fetch([key], d)
> > > + filename = fetcher.localpath(key)
> > > + d.appendVar("APTKEYFILES", " %s" % filename)
> > > }
> > >
> > > def aggregate_files(d, file_list, file_out):
> > > @@ -174,13 +169,17 @@ def get_distro_components_argument(d,
> > > is_host): else:
> > > return ""
> > >
> > > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > > +
> > > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> > > do_generate_keyring[dirs] = "${DL_DIR}"
> > > do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> > > do_generate_keyring() {
> > > if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > > + chmod 777 "${APTKEYTMPDIR}"
> > > for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > > - gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > > - --no-tty --homedir "${DL_DIR}" --import "$keyfile"
> > > + cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename "$keyfile")"
> > > + sudo apt-key add "$keyfile"
> > > done
> > > fi
> > > }
> > > @@ -222,7 +221,6 @@ isar_bootstrap() {
> > > if [ ${IS_HOST} ]; then
> > > ${DEBOOTSTRAP} $debootstrap_args \
> > > ${@get_distro_components_argument(d,
> > > True)} \
> > > - ${DEBOOTSTRAP_KEYRING} \
> > > "${@get_distro_suite(d, True)}" \
> > > "${ROOTFSDIR}" \
> > > "${@get_distro_source(d, True)}"
> > > @@ -231,7 +229,6 @@ isar_bootstrap() {
> > > "${DEBOOTSTRAP}" $debootstrap_args \
> > > --arch="${DISTRO_ARCH}" \
> > > ${@get_distro_components_argument(d,
> > > False)} \
> > > - ${DEBOOTSTRAP_KEYRING} \
> > > "${@get_distro_suite(d, False)}"
> > > \ "${ROOTFSDIR}" \
> > > "${@get_distro_source(d,
> > > False)}"
> >
>
--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant
Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
^ permalink raw reply [flat|nested] 16+ messages in thread
* Re: [PATCH v2 1/3] Fix and simplify apt keyring generation
2019-03-01 10:18 ` Henning Schild
@ 2019-03-06 16:29 ` Andreas Reichel
0 siblings, 0 replies; 16+ messages in thread
From: Andreas Reichel @ 2019-03-06 16:29 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On Fri, Mar 01, 2019 at 11:18:48AM +0100, Henning Schild wrote:
> Am Fri, 1 Mar 2019 10:09:49 +0100
> schrieb Andreas Reichel <andreas.reichel.ext@siemens.com>:
>
> > On Wed, Feb 27, 2019 at 05:12:42PM +0100, Henning Schild wrote:
> > > Am Wed, 27 Feb 2019 16:18:54 +0100
> > > schrieb "[ext] Andreas J. Reichel"
> > > <andreas.reichel.ext@siemens.com>:
> > > > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > >
> > > > Different fetcher stored keys in different sub dirs, and we can
> > > > never be sure about where downloaded files go.
> > > > To avoid this without making any assumptions, we ask the fetcher
> > > > where the file will be after it is downloaded. This way we also
> > > > don't need to parse the URL manually.
> > > >
> > > > The code is simplified by removing duplicate code and using
> > > > apt-key instead of manually calling gpg.
> > > >
> > > > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > > > ---
> > > > meta/classes/isar-bootstrap-helper.bbclass | 11 ++++++
> > > > .../isar-bootstrap/isar-bootstrap.inc | 39
> > > > +++++++++---------- 2 files changed, 29 insertions(+), 21
> > > > deletions(-)
> > > >
> > > > diff --git a/meta/classes/isar-bootstrap-helper.bbclass
> > > > b/meta/classes/isar-bootstrap-helper.bbclass index
> > > > d780b85..b8c41f9 100644 ---
> > > > a/meta/classes/isar-bootstrap-helper.bbclass +++
> > > > b/meta/classes/isar-bootstrap-helper.bbclass @@ -119,6 +119,16 @@
> > > > setup_root_file_system() { export LANG=C
> > > > export LANGUAGE=C
> > > > export LC_ALL=C
> > > > +
> > > > + if [ -d ${TMPDIR}/aptkeys ]; then
> > > > + for keyfile in ${TMPDIR}/aptkeys/*
> > > > + do
> > > > + kfn="tmp/$(basename $keyfile)"
> > > > + cp $keyfile "$ROOTFSDIR/$kfn"
> > > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key add
> > > > "/$kfn"
> > >
> > > Would probably be a good idea to work in /tmp/ as well and in a
> > > subdir. Try naming your key "root" or "usr" ...
> > >
> > > > + rm "$ROOTFSDIR/$kfn"
> > > > + done
> > > > + fi
> > > > sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-get update \
> > > > -o Dir::Etc::sourcelist="sources.list.d/isar-apt.list" \
> > > > -o Dir::Etc::sourceparts="-" \
> > > > @@ -128,6 +138,7 @@ setup_root_file_system() {
> > > > sudo -E chroot "$ROOTFSDIR" /usr/bin/dpkg
> > > > --add-architecture ${DISTRO_ARCH} sudo -E chroot
> > > > "$ROOTFSDIR" /usr/bin/apt-get update fi
> > > > + sudo -E chroot "$ROOTFSDIR" /usr/bin/apt-key update
> > > > sudo -E chroot "$ROOTFSDIR" \
> > > > /usr/bin/apt-get ${APT_ARGS} --download-only $PACKAGES \
> > > > ${IMAGE_TRANSIENT_PACKAGES}
> > > > diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
> > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc index
> > > > 234d339..2ef3b1e 100644 ---
> > > > a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc +++
> > > > b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc @@ -23,35
> > > > +23,30 @@ APTSRCS = "${WORKDIR}/apt-sources" APTSRCS_INIT =
> > > > "${WORKDIR}/apt-sources-init" BASEAPTSRCS =
> > > > "${WORKDIR}/base-apt-sources" APTKEYFILES = ""
> > > > -APTKEYRING = "${WORKDIR}/apt-keyring.gpg"
> > > > -DEBOOTSTRAP_KEYRING = ""
> > > > DEPLOY_ISAR_BOOTSTRAP ?= ""
> > > > -DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales"
> > > > +DISTRO_BOOTSTRAP_BASE_PACKAGES = "locales gnupg2
> > > > apt-transport-https ca-certificates"
> > >
> > > That looks more like leftovers from an experiment. Are you sure
> > > that is needed?
> >
> > This is needed because debootstrap can switch to https by itself if it
> > is unsure about the sources (apt-transport-https, ca-certificates),
> > gnupg2 we need for apt-key.
>
> Adding the packages should probably depend on whether DISTRO_APT_KEYS
> is non-empty. And adding the ones that the https-detector adds should
> be tested, not that dual mention makes debootstrap crap out.
>
> I guess the OVERRIDES method from 9196b820636a9 should be used for gpg
> support as well.
>
Done in v3.
> Henning
>
> > >
> > >
> > > In general i think that patch should be splittet into two or
> > > even more patches. Because it solves at least two problems.
> > >
> > > Henning
> > >
> > > > DISTRO_APT_PREMIRRORS ?= "${@ "http://ftp\.(\S+\.)?debian.org
> > > > file:///${REPO_BASE_DIR} \n" if
> > > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')) else
> > > > "" }" inherit base-apt-helper
> > > > python () {
> > > > - from urllib.parse import urlparse
> > > > distro_apt_keys = d.getVar("DISTRO_APT_KEYS", False)
> > > > - wd = d.getVar("WORKDIR", True)
> > > > + aptkeys = []
> > > > +
> > > > if distro_apt_keys:
> > > > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > > > ${APTKEYRING}")
> > > > - for key in distro_apt_keys.split():
> > > > - url = urlparse(key)
> > > > - filename = ''.join([wd, url.path])
> > > > - d.appendVar("SRC_URI", " %s" % key)
> > > > - d.appendVar("APTKEYFILES", " %s" % filename)
> > > > + aptkeys += distro_apt_keys.split()
> > > > +
> > > > if
> > > > bb.utils.to_boolean(d.getVar('ISAR_USE_CACHED_BASE_REPO')):
> > > > own_pub_key = d.getVar("BASE_REPO_KEY", False) if own_pub_key:
> > > > - d.setVar("DEBOOTSTRAP_KEYRING", "--keyring
> > > > ${APTKEYRING}")
> > > > - for key in own_pub_key.split():
> > > > - url = urlparse(key)
> > > > - filename = ''.join([wd, url.path])
> > > > - d.appendVar("SRC_URI", " %s" % key)
> > > > - d.appendVar("APTKEYFILES", " %s" % filename)
> > > > + aptkeys += own_pub_keys.split()
> > > > +
> > > > + for key in aptkeys:
> > > > + d.appendVar("SRC_URI", " %s" % key)
> > > > + fetcher = bb.fetch2.Fetch([key], d)
> > > > + filename = fetcher.localpath(key)
> > > > + d.appendVar("APTKEYFILES", " %s" % filename)
> > > > }
> > > >
> > > > def aggregate_files(d, file_list, file_out):
> > > > @@ -174,13 +169,17 @@ def get_distro_components_argument(d,
> > > > is_host): else:
> > > > return ""
> > > >
> > > > +APTKEYTMPDIR := "${TMPDIR}/aptkeys"
> > > > +
> > > > +do_generate_keyring[cleandirs] = "${APTKEYTMPDIR}"
> > > > do_generate_keyring[dirs] = "${DL_DIR}"
> > > > do_generate_keyring[vardeps] += "DISTRO_APT_KEYS"
> > > > do_generate_keyring() {
> > > > if [ -n "${@d.getVar("APTKEYFILES", True) or ""}" ]; then
> > > > + chmod 777 "${APTKEYTMPDIR}"
> > > > for keyfile in ${@d.getVar("APTKEYFILES", True)}; do
> > > > - gpg --no-default-keyring --keyring "${APTKEYRING}" \
> > > > - --no-tty --homedir "${DL_DIR}" --import
> > > > "$keyfile"
> > > > + cp "$keyfile" "${APTKEYTMPDIR}"/"$(basename
> > > > "$keyfile")"
> > > > + sudo apt-key add "$keyfile"
> > > > done
> > > > fi
> > > > }
> > > > @@ -222,7 +221,6 @@ isar_bootstrap() {
> > > > if [ ${IS_HOST} ]; then
> > > > ${DEBOOTSTRAP} $debootstrap_args \
> > > > ${@get_distro_components_argument(d,
> > > > True)} \
> > > > - ${DEBOOTSTRAP_KEYRING} \
> > > > "${@get_distro_suite(d, True)}" \
> > > > "${ROOTFSDIR}" \
> > > > "${@get_distro_source(d, True)}"
> > > > @@ -231,7 +229,6 @@ isar_bootstrap() {
> > > > "${DEBOOTSTRAP}" $debootstrap_args \
> > > > --arch="${DISTRO_ARCH}" \
> > > > ${@get_distro_components_argument(d,
> > > > False)} \
> > > > - ${DEBOOTSTRAP_KEYRING} \
> > > > "${@get_distro_suite(d,
> > > > False)}" \ "${ROOTFSDIR}" \
> > > > "${@get_distro_source(d,
> > > > False)}"
> > >
> >
>
--
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant
Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082
^ permalink raw reply [flat|nested] 16+ messages in thread
end of thread, other threads:[~2019-03-06 16:31 UTC | newest]
Thread overview: 16+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-27 15:18 [PATCH v2 0/3] Fixes usage of additional apt keys Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 1/3] Fix and simplify apt keyring generation Andreas J. Reichel
2019-02-27 16:12 ` Henning Schild
2019-03-01 9:09 ` Andreas Reichel
2019-03-01 10:18 ` Henning Schild
2019-03-06 16:29 ` Andreas Reichel
2019-03-01 9:16 ` Andreas Reichel
2019-03-01 10:11 ` Henning Schild
2019-03-01 10:22 ` Henning Schild
2019-03-06 16:29 ` Andreas Reichel
2019-02-27 15:18 ` [PATCH v2 2/3] Use all source lists in target root apt Andreas J. Reichel
2019-02-27 15:18 ` [PATCH v2 3/3] Separate apt-key entries from default keyring Andreas J. Reichel
2019-02-27 16:14 ` Henning Schild
2019-03-01 9:08 ` Andreas Reichel
2019-02-27 16:15 ` [PATCH v2 0/3] Fixes usage of additional apt keys Henning Schild
2019-03-01 9:17 ` Andreas Reichel
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox