Re: [PATCH v9 0/5] Fix usage of additional apt keys and repos

[Andreas Reichel <andreas.reichel.ext@siemens.com>]

Hi Maxim,

could reproduce it. Fix is on the way. It is a one-liner fix.

Regards
Andreas

On Mon, Apr 22, 2019 at 02:22:24PM +0200, Maxim Yu. Osipov wrote:
> Hi Andreas,
> 
> I've tested your series (with the docker use-case example you described in
> last patch in series). It works as described in the default case - without
> local apt caching enabled.
> 
> I've tested it with with signed local apt caching feature enabled.
> 
> The first stage - creation of local repo passed OK -
> 
> bitbake -c cache_base_repo multiconfig:qemuarm64-stretch:isar-image-base
> 
> But on the second stage the build failed (see log below).
> 
> I've double checked 'signed local apt caching feature' works fine in the
> current 'next'.
> 
> My local.conf is attached for convenience.
> 
> Regards,
> Maxim.
> 
> =============
> 
>  bitbake multiconfig:qemuarm64-stretch:isar-image-base
> Parsing recipes: 100% |#######################################################################################################################################################################################################|
> Time: 0:00:03
> Parsing of 26 .bb files complete (0 cached, 26 parsed). 390 targets, 0
> skipped, 0 masked, 0 errors.
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> NOTE: Resolving any missing task queue dependencies
> Initialising tasks: 100% |####################################################################################################################################################################################################|
> Time: 0:00:04
> NOTE: Executing RunQueue Tasks
> ERROR: mc:qemuarm64-stretch:isar-bootstrap-host-1.0-r0 do_bootstrap:
> Function failed: do_bootstrap (log file is located at /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.26651)
> ERROR: mc:qemuarm64-stretch:isar-bootstrap-target-1.0-r0 do_bootstrap:
> Function failed: do_bootstrap (log file is located at /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/temp/log.do_bootstrap.26652)
> ERROR: Logfile of failure stored in: /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.26651
> Log data follows:
> | DEBUG: Executing shell function do_bootstrap
> | W: Target architecture is the same as host architecture; disabling QEMU
> support
> | I: Running command: debootstrap --arch amd64 --verbose --variant=minbase
> --include=locales,gnupg2 --components=main,contrib,non-free stretch /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/rootfs file:////home/myo/work/isar/src/trunk/isar/build/downloads/base-apt/apt/debian
> | I: Retrieving InRelease
> | I: Checking Release signature
> | E: Release signed by unknown key (key id 75CB2BE443564A84)
> | WARNING: exit code 1 from a shell command.
> | ERROR: Function failed: do_bootstrap (log file is located at /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-host-debian-stretch-amd64/temp/log.do_bootstrap.26651)
> ERROR: Task (multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap)
> failed with exit code '1'
> ERROR: Logfile of failure stored in: /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/temp/log.do_bootstrap.26652
> Log data follows:
> | DEBUG: Executing shell function do_bootstrap
> | I: Running command: debootstrap --arch arm64 --foreign --verbose
> --variant=minbase
> --include=locales,gnupg2,apt-transport-https,ca-certificates
> --components=main,contrib,non-free stretch /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/rootfs file:////home/myo/work/isar/src/trunk/isar/build/downloads/base-apt/apt/debian
> | I: Retrieving InRelease
> | I: Checking Release signature
> | E: Release signed by unknown key (key id 75CB2BE443564A84)
> | WARNING: exit code 1 from a shell command.
> | ERROR: Function failed: do_bootstrap (log file is located at /home/myo/work/isar/src/trunk/isar/build/tmp/work/debian-stretch-arm64/isar-bootstrap-target/temp/log.do_bootstrap.26652)
> ERROR: Task (multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap)
> failed with exit code '1'
> NOTE: Tasks Summary: Attempted 53 tasks of which 0 didn't need to be rerun
> and 2 failed.
> 
> Summary: 2 tasks failed:
> 
> multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-host.bb:do_bootstrap
> 
> multiconfig:qemuarm64-stretch:/home/myo/work/isar/src/trunk/isar/meta/recipes-core/isar-bootstrap/isar-bootstrap-target.bb:do_bootstrap
> Summary: There were 2 ERROR messages shown, returning a non-zero exit code.
> 
> ====
> 
> 
> 
> 
> 
> On 4/16/19 3:05 PM, Andreas J. Reichel wrote:
> > From: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > Diff to v8: No revert, improve docs. Map changes for missing
> > raspbian-jessie.conf to raspbian-stretch.conf.
> > 
> > Last test of BASE_REPO_KEY worked before trivial rebasing.
> > 
> > Signed-off-by: Andreas Reichel <andreas.reichel.ext@siemens.com>
> > 
> > Andreas Reichel (5):
> >    Simplify and enhance apt-keyring generator
> >    Use apt-key to generate keyrings
> >    If we use a custom keyring debootstrap may fall to https
> >    raspbian-stretch: Use DISTRO_BOOTSTRAP_KEYS
> >    docs: Update user_manual.md
> > 
> >   doc/user_manual.md                            | 41 +++++++-
> >   meta-isar/conf/distro/raspbian-stretch.conf   |  2 +-
> >   .../conf/multiconfig/qemuamd64-buster.conf    |  1 -
> >   meta/conf/bitbake.conf                        |  1 +
> >   .../isar-bootstrap/isar-bootstrap-host.bb     |  4 +-
> >   .../isar-bootstrap/isar-bootstrap-target.bb   |  4 +-
> >   .../isar-bootstrap/isar-bootstrap.inc         | 95 +++++++++++++------
> >   7 files changed, 114 insertions(+), 34 deletions(-)
> > 
> 
> 
> -- 
> Maxim Osipov
> ilbers GmbH
> Maria-Merian-Str. 8
> 85521 Ottobrunn
> Germany
> +49 (151) 6517 6917
> mosipov@ilbers.de
> http://ilbers.de/
> Commercial register Munich, HRB 214197
> General Manager: Baurzhan Ismagulov

> #
> # This file is your local configuration file and is where all local user settings
> # are placed. The comments in this file give some guide to the options a new user
> # to the system might want to change but pretty much any configuration option can
> # be set in this file. More adventurous users can look at local.conf.extended
> # which contains other examples of configuration which can be placed in this file
> # but new users likely won't need any of them initially.
> #
> # Lines starting with the '#' character are commented out and in some cases the
> # default values are provided as comments to show people example syntax. Enabling
> # the option is a question of removing the # character and making any change to the
> # variable as required.
> 
> #
> # Machine Selection
> #
> # You need to select a specific machine to target the build with. There are a selection
> # of emulated machines available which can boot and run in the QEMU emulator:
> #
> # This sets the default machine to be qemuarm if no other machine is selected:
> MACHINE ??= "qemuarm"
> 
> #
> # Isar Configuration Selection
> #
> # You need to select a specific distribution configuration which will used for both:
> # generation of buildchroot environment and target root filesystem.
> #
> # This sets the default distribution configuration:
> DISTRO ??= "debian-stretch"
> DISTRO_ARCH ??= "armhf"
> 
> #
> # Multiple Configuration Selection
> #
> # If you want to use multiple configuration files for the build, list them in the
> # following option.
> #
> # This sets the default multiple configurations used:
> BBMULTICONFIG = " \
>     qemuarm-stretch \
>     qemuarm-buster \
>     qemuarm64-stretch \
>     qemuarm64-buster \
>     qemui386-stretch \
>     qemui386-buster \
>     qemuamd64-stretch \
>     bananapi-stretch \
>     de0-nano-soc-stretch \
>     hikey-stretch \
>     qemuamd64-buster \
>     qemuamd64-buster-tgz \
>     nand-ubi-demo-buster \
>     rpi-stretch \
> "
> 
> #
> # Where to place downloads
> #
> # During a first build the system will download many different source code tarballs
> # from various upstream projects. This can take a while, particularly if your network
> # connection is slow. These are all stored in DL_DIR. When wiping and rebuilding you
> # can preserve this directory to speed up this part of subsequent builds. This directory
> # is safe to share between multiple builds on the same machine too.
> #
> # The default is a downloads directory under TOPDIR which is the build directory.
> #
> #DL_DIR ?= "${TOPDIR}/downloads"
> 
> #
> # Where to place shared-state files
> #
> # BitBake has the capability to accelerate builds based on previously built output.
> # This is done using "shared state" files which can be thought of as cache objects
> # and this option determines where those files are placed.
> #
> # You can wipe out TMPDIR leaving this directory intact and the build would regenerate
> # from these files if no changes were made to the configuration. If changes were made
> # to the configuration, only shared state files where the state was still valid would
> # be used (done using checksums).
> #
> # The default is a sstate-cache directory under TOPDIR.
> #
> #SSTATE_DIR ?= "${TOPDIR}/sstate-cache"
> 
> #
> # Where to place the build output
> #
> # This option specifies where the bulk of the building work should be done and
> # where BitBake should place its temporary files and output. Keep in mind that
> # this includes the extraction and compilation of many applications and the toolchain
> # which can use Gigabytes of hard disk space.
> #
> # The default is a tmp directory under TOPDIR.
> #
> #TMPDIR = "${TOPDIR}/tmp"
> 
> #
> # Interactive shell configuration
> #
> # Under certain circumstances the system may need input from you and to do this it
> # can launch an interactive shell. It needs to do this since the build is
> # multithreaded and needs to be able to handle the case where more than one parallel
> # process may require the user's attention. The default is iterate over the available
> # terminal types to find one that works.
> #
> # Examples of the occasions this may happen are when resolving patches which cannot
> # be applied, to use the devshell or the kernel menuconfig
> #
> # Supported values are auto, gnome, xfce, rxvt, screen, konsole (KDE 3.x only), none
> # Note: currently, Konsole support only works for KDE 3.x due to the way
> # newer Konsole versions behave
> #OE_TERMINAL = "auto"
> # By default disable interactive patch resolution (tasks will just fail instead):
> PATCHRESOLVE = "noop"
> 
> #
> # Disk Space Monitoring during the build
> #
> # Monitor the disk space during the build. If there is less that 1GB of space or less
> # than 100K inodes in any key build location (TMPDIR, DL_DIR, SSTATE_DIR), gracefully
> # shutdown the build. If there is less that 100MB or 1K inodes, perform a hard abort
> # of the build. The reason for this is that running completely out of space can corrupt
> # files and damages the build in ways which may not be easily recoverable.
> # It's necesary to monitor /tmp, if there is no space left the build will fail
> # with very exotic errors.
> BB_DISKMON_DIRS = "\
>     STOPTASKS,${TMPDIR},1G,100K \
>     STOPTASKS,${DL_DIR},1G,100K \
>     STOPTASKS,${SSTATE_DIR},1G,100K \
>     STOPTASKS,/tmp,100M,100K \
>     ABORT,${TMPDIR},100M,1K \
>     ABORT,${DL_DIR},100M,1K \
>     ABORT,${SSTATE_DIR},100M,1K \
>     ABORT,/tmp,10M,1K"
> 
> #
> # Shared-state files from other locations
> #
> # As mentioned above, shared state files are prebuilt cache data objects which can
> # used to accelerate build time. This variable can be used to configure the system
> # to search other mirror locations for these objects before it builds the data itself.
> #
> # This can be a filesystem directory, or a remote url such as http or ftp. These
> # would contain the sstate-cache results from previous builds (possibly from other
> # machines). This variable works like fetcher MIRRORS/PREMIRRORS and points to the
> # cache locations to check for the shared objects.
> # NOTE: if the mirror uses the same structure as SSTATE_DIR, you need to add PATH
> # at the end as shown in the examples below. This will be substituted with the
> # correct path within the directory structure.
> #SSTATE_MIRRORS ?= "\
> #file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
> #file://.* file:///some/local/dir/sstate/PATH"
> 
> # CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to
> # track the version of this file when it was generated. This can safely be ignored if
> # this doesn't mean anything to you.
> CONF_VERSION = "1"
> 
> #
> # The default list of extra packages to be installed.
> IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck"
> 
> IMAGE_PREINSTALL += "docker-ce"
> THIRD_PARTY_APT_KEYS_append = " https://download.docker.com/linux/debian/gpg;md5sum=1afae06b34a13c1b3d9cb61a26285a15"
> DISTRO_APT_SOURCES_append = " conf/distro/docker-stretch.list"
> 
> BASE_REPO_KEY = "file:///home/myo/my_pub_key.key"
> 
> #
> # Enable cross-compilation support
> # NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
> ISAR_CROSS_COMPILE ?= "1"
> 
> #
> # Uncomment this to enable use of cached base repository
> ISAR_USE_CACHED_BASE_REPO ?= "1"
> 
> # Set root password to 'root'
> # Password was encrypted using following command:
> #   mkpasswd -m sha512crypt -R 10000
> # mkpasswd is part of the 'whois' package of Debian
> CFG_ROOT_PW ?= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"


-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082