Re: [PATCH v4 0/8] pre-processing pipeline and transient package replacement

["Maxim Yu. Osipov" <mosipov@ilbers.de>]

[-- Attachment #1: Type: text/plain, Size: 14310 bytes --]

Hi Claudius,

On 5/27/19 10:24 AM, Claudius Heine wrote:
> Hi Maxim,
> 
> On 27/05/2019 10.10, Maxim Yu. Osipov wrote:
>> Hi Claudius,
>>
>> On 5/27/19 9:56 AM, Claudius Heine wrote:
>>> Hi Maxim,
>>>
>>> On 24/05/2019 14.56, Maxim Yu. Osipov wrote:
>>>> Hi Claudius,
>>>>
>>>> CI passed OK.
>>>>
>>>> But when I tried to manually test SDK creation procedure (as exactly 
>>>> described in user_manual) it failed:
>>>
>>> Have you tested if it does not fail the same way in next when using 
>>> the base-apt repository?
>>
>> I've performed the following steps (actually exactly followed current 
>> SDK creation description in user_manual):
>>
>> 1) In clean tree with your patches applied, I've enabled cross 
>> compilation in local.conf and launched SDK creation procedure for 
>> qemuarm-stretch
>> 'build bitbake -c do_populate_sdk 
>> multiconfig:qemuarm-stretch:isar-image-base'
>>
>> The build was OK (as expected because such SDK creation build passed 
>> in CI).
>>
>> 2) I've mounted needed directories mount_chroot.sh 
>> sdk-debian-stretch-armhf and chrooted into in.
>>
>> 3) In SDK chroot commands apt-get update and apt-get install 
>> libhello-dev:armhf failed (logs are below).
>>
>>
>> I've performed the same procedure w/o your patches - it works as 
>> described in user_manual.
> 
> Well that does not answer my question wherever or not you have enabled 
> ISAR_USE_CACHED_BASE_REPO, in both builds. The logs mention 
> 'file:/base-apt/debian' so I am reasonable sure that you had 
> `ISAR_USE_CACHED_BASE_REPO ="1"` and I just tried that with a filled 
> base-apt in current next resulting in the same error.

You are right - I've just double checked from the scratch (my current 
local.conf is attached) - the only thing now is that resolv.conf is 
missing to sdkchroot, so 'apt-get update' complains (see below). When I 
copied resolv.conf from my host to sdk chroot 'apt-get update/apt-get 
install libhello-dev:armhf' in chroot succeeded.

It's not a bug as sdk chroot may be distributed to other machines (or PC 
can be connected later to other network) and resolv.conf has to be 
substituted accordingly.

I'll mention this explicitly in user manual.

root@tiberius:/# dpkg -l | grep essen
ii  crossbuild-essential-armhf           12.3                   all 
     Informational list of cross-build-essential packages
root@tiberius:/# apt-get update
Get:1 file:/isar-apt isar InRelease
Ign:1 file:/isar-apt isar InRelease
Get:2 file:/isar-apt isar Release [3,569 B]
Err:3 http://ftp.debian.org/debian stretch InRelease
   Temporary failure resolving 'ftp.debian.org'
Err:4 http://ftp.debian.org/debian stretch-updates InRelease
   Temporary failure resolving 'ftp.debian.org'
Err:5 http://security.debian.org stretch/updates InRelease
   Temporary failure resolving 'security.debian.org'
Get:2 file:/isar-apt isar Release [3,569 B]
Get:6 file:/isar-apt isar Release.gpg
Ign:6 file:/isar-apt isar Release.gpg
Get:7 file:/isar-apt isar/main armhf Packages [1,666 B]
Reading package lists... Done
W: Failed to fetch http://ftp.debian.org/debian/dists/stretch/InRelease 
Temporary failure resolving 'ftp.debian.org'
W: Failed to fetch 
http://ftp.debian.org/debian/dists/stretch-updates/InRelease  Temporary 
failure resolving 'ftp.debian.org'
W: Failed to fetch 
http://security.debian.org/dists/stretch/updates/InRelease  Temporary 
failure resolving 'security.debian.org'
W: Some index files failed to download. They have been ignored, or old 
ones used instead.

ls -l /etc/resolv.conf
ls: cannot access '/etc/resolv.conf': No such file or directory
root@tiberius:/#


Regards,
Maxim.



>>
>> Regards,
>> Maxim.
>>
>>
>>> regards,
>>> Claudius
>>>
>>>>
>>>> I've mounted needed directories with
>>>> 'mount_chroot.sh sdk-debian-stretch-armhf
>>>>
>>>> and chroot'ed into sdk:
>>>>
>>>> root@tiberius:/#  apt-get install libhello-dev:armhf
>>>> Reading package lists... Done
>>>> Building dependency tree
>>>> Reading state information... Done
>>>> The following additional packages will be installed:
>>>>    gcc-6-base:armhf libc6:armhf libgcc1:armhf libhello:armhf
>>>> Suggested packages:
>>>>    glibc-doc:armhf locales:armhf
>>>> The following NEW packages will be installed:
>>>>    gcc-6-base:armhf libc6:armhf libgcc1:armhf libhello:armhf 
>>>> libhello-dev:armhf
>>>> 0 upgraded, 5 newly installed, 0 to remove and 0 not upgraded.
>>>> Need to get 0 B/2,564 kB of archives.
>>>> After this operation, 8,987 kB of additional disk space will be used.
>>>> Do you want to continue? [Y/n]
>>>> Get:1 file:/base-apt/debian stretch/main armhf gcc-6-base armhf 
>>>> 6.3.0-18+deb9u1 [180 kB]
>>>> Err:1 file:/base-apt/debian stretch/main armhf gcc-6-base armhf 
>>>> 6.3.0-18+deb9u1
>>>>    File not found - 
>>>> /base-apt/debian/pool/main/g/gcc-6/gcc-6-base_6.3.0-18+deb9u1_armhf.deb 
>>>> (2: No such file or directory)
>>>> Get:2 file:/base-apt/debian stretch/main armhf libgcc1 armhf 
>>>> 1:6.3.0-18+deb9u1 [38.5 kB]
>>>> Err:2 file:/base-apt/debian stretch/main armhf libgcc1 armhf 
>>>> 1:6.3.0-18+deb9u1
>>>>    File not found - 
>>>> /base-apt/debian/pool/main/g/gcc-6/libgcc1_6.3.0-18+deb9u1_armhf.deb 
>>>> (2: No such file or directory)
>>>> Get:3 file:/base-apt/debian stretch/main armhf libc6 armhf 
>>>> 2.24-11+deb9u4 [2,341 kB]
>>>> Err:3 file:/base-apt/debian stretch/main armhf libc6 armhf 
>>>> 2.24-11+deb9u4
>>>>    File not found - 
>>>> /base-apt/debian/pool/main/g/glibc/libc6_2.24-11+deb9u4_armhf.deb 
>>>> (2: No such file or directory)
>>>> Get:4 file:/isar-apt isar/main armhf libhello armhf 0.1 [2,826 B]
>>>> Get:5 file:/isar-apt isar/main armhf libhello-dev armhf 0.1 [2,418 B]
>>>> E: Failed to fetch 
>>>> file:/base-apt/debian/pool/main/g/gcc-6/gcc-6-base_6.3.0-18+deb9u1_armhf.deb 
>>>>   File not found - 
>>>> /base-apt/debian/pool/main/g/gcc-6/gcc-6-base_6.3.0-18+deb9u1_armhf.deb 
>>>> (2: No such file or directory)
>>>> E: Failed to fetch 
>>>> file:/base-apt/debian/pool/main/g/gcc-6/libgcc1_6.3.0-18+deb9u1_armhf.deb 
>>>>  File not found - 
>>>> /base-apt/debian/pool/main/g/gcc-6/libgcc1_6.3.0-18+deb9u1_armhf.deb 
>>>> (2: No such file or directory)
>>>> E: Failed to fetch 
>>>> file:/base-apt/debian/pool/main/g/glibc/libc6_2.24-11+deb9u4_armhf.deb 
>>>> File not found - 
>>>> /base-apt/debian/pool/main/g/glibc/libc6_2.24-11+deb9u4_armhf.deb 
>>>> (2: No such file or directory)
>>>> E: Unable to fetch some archives, maybe run apt-get update or try 
>>>> with --fix-missing?
>>>> root@tiberius:/# apt-get update
>>>> Get:1 file:/base-apt/debian stretch InRelease
>>>> Ign:1 file:/base-apt/debian stretch InRelease
>>>> Get:2 file:/isar-apt isar InRelease
>>>> Ign:2 file:/isar-apt isar InRelease
>>>> Get:3 file:/base-apt/debian stretch Release
>>>> Ign:3 file:/base-apt/debian stretch Release
>>>> Get:4 file:/isar-apt isar Release [3,584 B]
>>>> Get:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Ign:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Get:6 file:/base-apt/debian stretch/main all Packages
>>>> Ign:6 file:/base-apt/debian stretch/main all Packages
>>>> Get:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Ign:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Get:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Ign:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Get:9 file:/base-apt/debian stretch/main Translation-en
>>>> Ign:9 file:/base-apt/debian stretch/main Translation-en
>>>> Get:4 file:/isar-apt isar Release [3,584 B]
>>>> Get:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Ign:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Get:6 file:/base-apt/debian stretch/main all Packages
>>>> Ign:6 file:/base-apt/debian stretch/main all Packages
>>>> Get:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Ign:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Get:10 file:/isar-apt isar Release.gpg
>>>> Ign:10 file:/isar-apt isar Release.gpg
>>>> Get:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Ign:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Get:9 file:/base-apt/debian stretch/main Translation-en
>>>> Ign:9 file:/base-apt/debian stretch/main Translation-en
>>>> Get:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Ign:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Get:6 file:/base-apt/debian stretch/main all Packages
>>>> Ign:6 file:/base-apt/debian stretch/main all Packages
>>>> Get:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Ign:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Get:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Ign:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Get:9 file:/base-apt/debian stretch/main Translation-en
>>>> Ign:9 file:/base-apt/debian stretch/main Translation-en
>>>> Get:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Ign:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Get:6 file:/base-apt/debian stretch/main all Packages
>>>> Ign:6 file:/base-apt/debian stretch/main all Packages
>>>> Get:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Ign:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Get:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Ign:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Get:9 file:/base-apt/debian stretch/main Translation-en
>>>> Ign:9 file:/base-apt/debian stretch/main Translation-en
>>>> Get:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Ign:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Get:6 file:/base-apt/debian stretch/main all Packages
>>>> Ign:6 file:/base-apt/debian stretch/main all Packages
>>>> Get:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Ign:7 file:/base-apt/debian stretch/main amd64 Packages
>>>> Get:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Ign:8 file:/base-apt/debian stretch/main Translation-en_US
>>>> Get:9 file:/base-apt/debian stretch/main Translation-en
>>>> Ign:9 file:/base-apt/debian stretch/main Translation-en
>>>> Get:5 file:/base-apt/debian stretch/main armhf Packages
>>>> Err:5 file:/base-apt/debian stretch/main armhf Packages
>>>>    File not found - 
>>>> /base-apt/debian/dists/stretch/main/binary-armhf/Packages (2: No 
>>>> such file or directory)
>>>> Get:6 file:/base-apt/debian stretch/main all Packages
>>>> Ign:6 file:/base-apt/debian stretch/main all Packages
>>>> Reading package lists... Done
>>>> E: Failed to fetch 
>>>> file:/base-apt/debian/dists/stretch/main/binary-armhf/Packages  File 
>>>> not found - 
>>>> /base-apt/debian/dists/stretch/main/binary-armhf/Packages (2: No 
>>>> such file or directory)
>>>> E: Some index files failed to download. They have been ignored, or 
>>>> old ones used instead.
>>>>
>>>>
>>>> Regards,
>>>> Maxim.
>>>>
>>>> On 5/23/19 4:55 PM, claudius.heine.ext@siemens.com wrote:
>>>>> From: Claudius Heine <ch@denx.de>
>>>>>
>>>>> Hi,
>>>>>
>>>>> I decided to heed Hennings advice and order the removal patch 
>>>>> before the split
>>>>> patch. This way the transient package removal patch becomes more 
>>>>> complete.
>>>>>
>>>>> regards,
>>>>> Claudius
>>>>>
>>>>> changes from v3:
>>>>>   - ordered the transient package removal patch before the split
>>>>>
>>>>> changes from v2:
>>>>>   - added 'copy-package-cache' to the ROOTFS_FEATURES in image.bbclass
>>>>>
>>>>> changes from v1:
>>>>>   - rebased to current next
>>>>>   - added some more documentation
>>>>>
>>>>> Claudius Heine (8):
>>>>>    meta: remove transient package support
>>>>>    split up isar-bootstrap helper and implement pre-process pipeline
>>>>>    meta/classes: add image-locales-extension class
>>>>>    meta/classes: add image-account-extension class
>>>>>    doc: update description of image customization
>>>>>    doc: some fixes
>>>>>    meta-isar: local.conf.sample: update root password and isar user
>>>>>      creation
>>>>>    RECIPE-API-CHANGELOG: update transient package removal + root 
>>>>> password
>>>>>
>>>>>   RECIPE-API-CHANGELOG.md                       |  22 ++
>>>>>   doc/user_manual.md                            |  53 +++-
>>>>>   meta-isar/conf/local.conf.sample              |  12 +-
>>>>>   .../recipes-app/example-raw/files/postinst    |  13 -
>>>>>   meta/classes/image-account-extension.bbclass  | 257 
>>>>> ++++++++++++++++++
>>>>>   meta/classes/image-locales-extension.bbclass  |  97 +++++++
>>>>>   meta/classes/image.bbclass                    |  30 +-
>>>>>   meta/classes/isar-bootstrap-helper.bbclass    | 143 ----------
>>>>>   meta/classes/rootfs.bbclass                   | 176 +++++++++++-
>>>>>   .../isar-bootstrap/isar-bootstrap.inc         |   5 +
>>>>>   .../buildchroot/buildchroot-host.bb           |  16 +-
>>>>>   .../buildchroot/buildchroot-target.bb         |   2 -
>>>>>   .../buildchroot/buildchroot.inc               |  26 +-
>>>>>   meta/recipes-devtools/sdkchroot/sdkchroot.bb  |  48 ++--
>>>>>   .../isar-cfg-localepurge/files/postinst       |  15 -
>>>>>   .../isar-cfg-localepurge.bb                   |  70 -----
>>>>>   .../isar-cfg-rootpw/files/postinst.tmpl       |  19 --
>>>>>   .../isar-cfg-rootpw/isar-cfg-rootpw.bb        |  19 --
>>>>>   18 files changed, 663 insertions(+), 360 deletions(-)
>>>>>   create mode 100644 meta/classes/image-account-extension.bbclass
>>>>>   create mode 100644 meta/classes/image-locales-extension.bbclass
>>>>>   delete mode 100644 meta/classes/isar-bootstrap-helper.bbclass
>>>>>   delete mode 100644 
>>>>> meta/recipes-support/isar-cfg-localepurge/files/postinst
>>>>>   delete mode 100644 
>>>>> meta/recipes-support/isar-cfg-localepurge/isar-cfg-localepurge.bb
>>>>>   delete mode 100644 
>>>>> meta/recipes-support/isar-cfg-rootpw/files/postinst.tmpl
>>>>>   delete mode 100644 
>>>>> meta/recipes-support/isar-cfg-rootpw/isar-cfg-rootpw.bb
>>>>>
>>>>
>>>>
>>>
>>
>>
> 


-- 
Maxim Osipov
ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn
Germany
+49 (151) 6517 6917
mosipov@ilbers.de
http://ilbers.de/
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

[-- Attachment #2: local.conf --]
[-- Type: text/plain, Size: 7313 bytes --]

#
# This file is your local configuration file and is where all local user settings
# are placed. The comments in this file give some guide to the options a new user
# to the system might want to change but pretty much any configuration option can
# be set in this file. More adventurous users can look at local.conf.extended
# which contains other examples of configuration which can be placed in this file
# but new users likely won't need any of them initially.
#
# Lines starting with the '#' character are commented out and in some cases the
# default values are provided as comments to show people example syntax. Enabling
# the option is a question of removing the # character and making any change to the
# variable as required.

#
# Machine Selection
#
# You need to select a specific machine to target the build with. There are a selection
# of emulated machines available which can boot and run in the QEMU emulator:
#
# This sets the default machine to be qemuarm if no other machine is selected:
MACHINE ??= "qemuarm"

#
# Isar Configuration Selection
#
# You need to select a specific distribution configuration which will used for both:
# generation of buildchroot environment and target root filesystem.
#
# This sets the default distribution configuration:
DISTRO ??= "debian-stretch"
DISTRO_ARCH ??= "armhf"

#
# Multiple Configuration Selection
#
# If you want to use multiple configuration files for the build, list them in the
# following option.
#
# This sets the default multiple configurations used:
BBMULTICONFIG = " \
    qemuarm-stretch \
    qemuarm-buster \
    qemuarm64-stretch \
    qemuarm64-buster \
    qemui386-stretch \
    qemui386-buster \
    qemuamd64-stretch \
    bananapi-stretch \
    de0-nano-soc-stretch \
    hikey-stretch \
    qemuamd64-buster \
    qemuamd64-buster-tgz \
    nand-ubi-demo-buster \
    rpi-stretch \
"

#
# Where to place downloads
#
# During a first build the system will download many different source code tarballs
# from various upstream projects. This can take a while, particularly if your network
# connection is slow. These are all stored in DL_DIR. When wiping and rebuilding you
# can preserve this directory to speed up this part of subsequent builds. This directory
# is safe to share between multiple builds on the same machine too.
#
# The default is a downloads directory under TOPDIR which is the build directory.
#
#DL_DIR ?= "${TOPDIR}/downloads"

#
# Where to place shared-state files
#
# BitBake has the capability to accelerate builds based on previously built output.
# This is done using "shared state" files which can be thought of as cache objects
# and this option determines where those files are placed.
#
# You can wipe out TMPDIR leaving this directory intact and the build would regenerate
# from these files if no changes were made to the configuration. If changes were made
# to the configuration, only shared state files where the state was still valid would
# be used (done using checksums).
#
# The default is a sstate-cache directory under TOPDIR.
#
#SSTATE_DIR ?= "${TOPDIR}/sstate-cache"

#
# Where to place the build output
#
# This option specifies where the bulk of the building work should be done and
# where BitBake should place its temporary files and output. Keep in mind that
# this includes the extraction and compilation of many applications and the toolchain
# which can use Gigabytes of hard disk space.
#
# The default is a tmp directory under TOPDIR.
#
#TMPDIR = "${TOPDIR}/tmp"

#
# Interactive shell configuration
#
# Under certain circumstances the system may need input from you and to do this it
# can launch an interactive shell. It needs to do this since the build is
# multithreaded and needs to be able to handle the case where more than one parallel
# process may require the user's attention. The default is iterate over the available
# terminal types to find one that works.
#
# Examples of the occasions this may happen are when resolving patches which cannot
# be applied, to use the devshell or the kernel menuconfig
#
# Supported values are auto, gnome, xfce, rxvt, screen, konsole (KDE 3.x only), none
# Note: currently, Konsole support only works for KDE 3.x due to the way
# newer Konsole versions behave
#OE_TERMINAL = "auto"
# By default disable interactive patch resolution (tasks will just fail instead):
PATCHRESOLVE = "noop"

#
# Disk Space Monitoring during the build
#
# Monitor the disk space during the build. If there is less that 1GB of space or less
# than 100K inodes in any key build location (TMPDIR, DL_DIR, SSTATE_DIR), gracefully
# shutdown the build. If there is less that 100MB or 1K inodes, perform a hard abort
# of the build. The reason for this is that running completely out of space can corrupt
# files and damages the build in ways which may not be easily recoverable.
# It's necesary to monitor /tmp, if there is no space left the build will fail
# with very exotic errors.
BB_DISKMON_DIRS = "\
    STOPTASKS,${TMPDIR},1G,100K \
    STOPTASKS,${DL_DIR},1G,100K \
    STOPTASKS,${SSTATE_DIR},1G,100K \
    STOPTASKS,/tmp,100M,100K \
    ABORT,${TMPDIR},100M,1K \
    ABORT,${DL_DIR},100M,1K \
    ABORT,${SSTATE_DIR},100M,1K \
    ABORT,/tmp,10M,1K"

#
# Shared-state files from other locations
#
# As mentioned above, shared state files are prebuilt cache data objects which can
# used to accelerate build time. This variable can be used to configure the system
# to search other mirror locations for these objects before it builds the data itself.
#
# This can be a filesystem directory, or a remote url such as http or ftp. These
# would contain the sstate-cache results from previous builds (possibly from other
# machines). This variable works like fetcher MIRRORS/PREMIRRORS and points to the
# cache locations to check for the shared objects.
# NOTE: if the mirror uses the same structure as SSTATE_DIR, you need to add PATH
# at the end as shown in the examples below. This will be substituted with the
# correct path within the directory structure.
#SSTATE_MIRRORS ?= "\
#file://.* http://someserver.tld/share/sstate/PATH;downloadfilename=PATH \n \
#file://.* file:///some/local/dir/sstate/PATH"

# CONF_VERSION is increased each time build/conf/ changes incompatibly and is used to
# track the version of this file when it was generated. This can safely be ignored if
# this doesn't mean anything to you.
CONF_VERSION = "1"

#
# The default list of extra packages to be installed.
IMAGE_INSTALL = "hello-isar example-raw example-module-${KERNEL_NAME} enable-fsck"

#
# Enable cross-compilation support
# NOTE: this works on build host >= stretch for armhf, arm64 and amd64 targets for now.
ISAR_CROSS_COMPILE ?= "1"

#
# Uncomment this to enable use of cached base repository
#ISAR_USE_CACHED_BASE_REPO ?= "1"

# Set root password to 'root'
# Password was encrypted using following command:
#   mkpasswd -m sha512crypt -R 10000
# mkpasswd is part of the 'whois' package of Debian
USERS += "root"
USER_root[password] ??= "$6$rounds=10000$RXeWrnFmkY$DtuS/OmsAS2cCEDo0BF5qQsizIrq6jPgXnwv3PHqREJeKd1sXdHX/ayQtuQWVDHe0KIO0/sVH8dvQm1KthF0d/"

GROUPS += "isar"
GROUP_isar[flags] = "system"

USERS += "isar"
USER_isar[gid] = "isar"
USER_isar[home] = "/var/lib/isar"
USER_isar[comment] = "My isar user"
USER_isar[flags] = "system create-home"