Re: [PATCH v9 0/5] Fix usage of additional apt keys and repos

["Maxim Yu. Osipov" <mosipov@ilbers.de>]

[-- Attachment #1: Type: text/plain, Size: 6700 bytes --]

Hi Andreas,

I've tested your series (with the docker use-case example you described 
in last patch in series). It works as described in the default case - 
without local apt caching enabled.

I've tested it with with signed local apt caching feature enabled.

The first stage - creation of local repo passed OK -

bitbake -c cache_base_repo multiconfig:qemuarm64-stretch:isar-image-base

But on the second stage the build failed (see log below).

I've double checked 'signed local apt caching feature' works fine in the 
current 'next'.

My local.conf is attached for convenience.

Regards,
Maxim.

=============

  bitbake multiconfig:qemuarm64-stretch:isar-image-base
Parsing recipes: 100% 
|#######################################################################################################################################################################################################| 
Time: 0:00:03
Parsing of 26 .bb files complete (0 cached, 26 parsed). 390 targets, 0 
skipped, 0 masked, 0 errors.
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
NOTE: Resolving any missing task queue dependencies
Initialising tasks: 100% 
|####################################################################################################################################################################################################| 
Time: 0:00:04
NOTE: Executing RunQueue Tasks
ERROR: mc:qemuarm64-stretch:isar-bootstrap-host-1.0-r0 do_bootstrap: 
Function failed: do_bootstrap (log file is located at 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.26651)
ERROR: mc:qemuarm64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap: 
Function failed: do_bootstrap (log file is located at 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/temp/log.do_bootstrap.26652)
ERROR: Logfile of failure stored in: 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.26651
Log data follows:
| DEBUG: Executing shell function do_bootstrap
| W: Target architecture is the same as host architecture; disabling 
QEMU support
| I: Running command: debootstrap --arch amd64 --verbose 
--variant=minbase --include=locales,gnupg2 
--components=main,contrib,non-free stretch 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/rootfs 
file:////home/myo/work/isar/src/trunk/isar/build/downloads/base-apt/apt/debian
| I: Retrieving InRelease
| I: Checking Release signature
| E: Release signed by unknown key (key id 75CB2BE443564A84)
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_bootstrap (log file is located at 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.26651)
ERROR: Task 
(multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap) 
failed with exit code '1'
ERROR: Logfile of failure stored in: 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/temp/log.do_bootstrap.26652
Log data follows:
| DEBUG: Executing shell function do_bootstrap
| I: Running command: debootstrap --arch arm64 --foreign --verbose 
--variant=minbase 
--include=locales,gnupg2,apt-transport-https,ca-certificates 
--components=main,contrib,non-free stretch 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs 
file:////home/myo/work/isar/src/trunk/isar/build/downloads/base-apt/apt/debian
| I: Retrieving InRelease
| I: Checking Release signature
| E: Release signed by unknown key (key id 75CB2BE443564A84)
| WARNING: exit code 1 from a shell command.
| ERROR: Function failed: do_bootstrap (log file is located at 
/home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/temp/log.do_bootstrap.26652)
ERROR: Task 
(multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap) 
failed with exit code '1'
NOTE: Tasks Summary: Attempted 53 tasks of which 0 didn't need to be 
rerun and 2 failed.

Summary: 2 tasks failed:
 
multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap
 
multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap
Summary: There were 2 ERROR messages shown, returning a non-zero exit code.

====





On 4/16/19 3:05 PM, Andreas J. Reichel wrote:
> From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Diff to v8: No revert, improve docs. Map changes for missing
> raspbian-jessie.conf to raspbian-stretch.conf.
> 
> Last test of BASE_REPO_KEY worked before trivial rebasing.
> 
> Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> 
> Andreas Reichel (5):
>    Simplify and enhance apt-keyring generator
>    Use apt-key to generate keyrings
>    If we use a custom keyring debootstrap may fall to https
>    raspbian-stretch: Use DISTRO_BOOTSTRAP_KEYS
>    docs: Update user_manual.md
> 
>   doc/user_manual.md                            | 41 +++++++-
>   meta-isar/conf/distro/raspbian-stretch.conf   |  2 +-
>   .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
>   meta/conf/bitbake.conf                        |  1 +
>   .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
>   .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
>   .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
>   7 files changed, 114 insertions(+), 34 deletions(-)
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[-- Attachment #2: local.conf --]
[-- Type: text/plain, Size: 7354 bytes --]

#
# This file is your local configuration file and is where all local user settings
# are placed. The comments in this file give some guide to the options a new user
# to the system might want to change but pretty much any configuration option can
# be set in this file. More adventurous users can look at local.conf.extended
# which contains other examples of configuration which can be placed in this file
# but new users likely won't need any of them initially.
#
# Lines starting with the '#' character are commented out and in some cases the
# default values are provided as comments to show people example syntax. Enabling
# the option is a question of removing the # character and making any change to the
# variable as required.

#
# Machine Selection
#
# You need to select a specific machine to target the build with. There are a selection
# of emulated machines available which can boot and run in the QEMU emulator:
#
# This sets the default machine to be qemuarm if no other machine is selected:
MACHINE ??= "qemuarm"

#
# Isar Configuration Selection
#
# You need to select a specific distribution configuration which will used for both:
# generation of buildchroot environment and target root filesystem.
#
# This sets the default distribution configuration:
DISTRO ??= "debian-stretch"
DISTRO_ARCH ??= "armhf"

#
# Multiple Configuration Selection
#
# If you want to use multiple configuration files for the build, list them in the
# following option.
#
# This sets the default multiple configurations used:
BBMULTICONFIG = " \
    qemuarm-stretch \
    qemuarm-buster \
    qemuarm64-stretch \
    qemuarm64-buster \
    qemui386-stretch \
    qemui386-buster \
    qemuamd64-stretch \
    bananapi-stretch \
    de0-nano-soc-stretch \
    hikey-stretch \
    qemuamd64-buster \
    qemuamd64-buster-tgz \
    nand-ubi-demo-buster \
    rpi-stretch \
"

#
# Where to place downloads
#
# During a first build the system will download many different source code tarballs
# from various upstream projects. This can take a while, particularly if your network
# connection is slow. These are all stored in DL_DIR. When wiping and rebuilding you
# can preserve this directory to speed up this part of subsequent builds. This directory
# is safe to share between multiple builds on the same machine too.
#
# The default is a downloads directory under TOPDIR which is the build directory.
#
#DL_DIR ?= "${TOPDIR}/downloads"

#
# Where to place shared-state files
#
# BitBake has the capability to accelerate builds based on previously built output.
# This is done using "shared state" files which can be thought of as cache objects
# and this option determines where those files are placed.
#
# You can wipe out TMPDIR leaving this directory intact and the build would regenerate
# from these files if no changes were made to the configuration. If changes were made
# to the configuration, only shared state files where the state was still valid would
# be used (done using checksums).
#
# The default is a sstate-cache directory under TOPDIR.
#
#SSTATE_DIR ?= "${TOPDIR}/sstate-cache"

#
# Where to place the build output
#
# This option specifies where the bulk of the building work should be done and
# where BitBake should place its temporary files and output. Keep in mind that
# this includes the extraction and compilation of many applications and the toolchain
# which can use Gigabytes of hard disk space.
#
# The default is a tmp directory under TOPDIR.
#
#TMPDIR = "${TOPDIR}/tmp"

#
# Interactive shell configuration
#
# Under certain circumstances the system may need input from you and to do this it
# can launch an interactive shell. It needs to do this since the build is
# multithreaded and needs to be able to handle the case where more than one parallel
# process may require the user's attention. The default is iterate over the available
# terminal types to find one that works.
#
# Examples of the occasions this may happen are when resolving patches which cannot
# be applied, to use the devshell or the kernel menuconfig
#
# Supported values are auto, gnome, xfce, rxvt, screen, konsole (KDE 3.x only), none
# Note: currently, Konsole support only works for KDE 3.x due to the way
# newer Konsole versions behave
#OE_TERMINAL = "auto"
# By default disable interactive patch resolution (tasks will just fail instead):
PATCHRESOLVE = "noop"

#
# Disk Space Monitoring during the build
#
# Monitor the disk space during the build. If there is less that 1GB of space or less
# than 100K inodes in any key build location (TMPDIR, DL_DIR, SSTATE_DIR), gracefully
# shutdown the build. If there is less that 100MB or 1K inodes, perform a hard abort
# of the build. The reason for this is that running completely out of space can corrupt
# files and damages the build in ways which may not be easily recoverable.
# It's necesary to monitor /tmp, if there is no space left the build will fail
# with very exotic errors.
BB_DISKMON_DIRS = "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K \
    ABORT,/tmp,10M,1K"

#
# Shared-state files from other locations
#
# As mentioned above, shared state files are prebuilt cache data objects which can
# used to accelerate build time. This variable can be used to configure the system
# to search other mirror locations for these objects before it builds the data itself.
#
# This can be a filesystem directory, or a remote url such as http or ftp. These
# would contain the sstate-cache results from previous builds (possibly from other
# machines). This variable works like fetcher MIRRORS/PREMIRRORS and points to the
# cache locations to check for the shared objects.
# NOTE: if the mirror uses the same structure as SSTATE_DIR, you need to add PATH
# at the end as shown in the examples below. This will be substituted with the
# correct path within the directory structure.
#SSTATE_MIRRORS ?= "\
#file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
#file://.* file:///some/local/dir/sstate/PATH"

# CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to
# track the version of this file when it was generated. This can safely be ignored if
# this doesn't mean anything to you.
CONF_VERSION = "1"

#
# The default list of extra packages to be installed.
IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck"

IMAGE_PREINSTALL += "docker-ce"
THIRD_PARTY_APT_KEYS_append = " https://download.docker.com/linux/debian/gpg;md5sum=1afae06b34a13c1b3d9cb61a26285a15"
DISTRO_APT_SOURCES_append = " conf/distro/docker-stretch.list"

BASE_REPO_KEY = "file:///home/myo/my_pub_key.key"

#
# Enable cross-compilation support
# NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
ISAR_CROSS_COMPILE ?= "1"

#
# Uncomment this to enable use of cached base repository
ISAR_USE_CACHED_BASE_REPO ?= "1"

# Set root password to 'root'
# Password was encrypted using following command:
#   mkpasswd -m sha512crypt -R 10000
# mkpasswd is part of the 'whois' package of Debian
CFG_ROOT_PW ?= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"