* [PATCH 0/6] Isar apt cache implementation
@ 2017-08-27 15:13 Alexander Smirnov
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
` (5 more replies)
0 siblings, 6 replies; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Hello all,
this is the first series that introduces apt caching feature. The full
implementation can be found in branch: lenormf/develop-l20170602-dpkg-cross
A few words about apt caching...
The key idea is to have possibility to create custom apt repositories
that could be published and shared between projects. So the packages are
built once, repo is created and the repo is used for further builds.
Also this repo could be used in already running systems via apt-get.
Another requirement for this feature is to follow Debian way, so
reprepro tool is used to manage binary cache.
The binary repo is introduced as a separate layer what simplifies
exporting of just-built repo and importing of existing repo.
The whole implementation is split into two steps to simplify the review:
1. Basic apt implementation that allows to create binary cache. The
feature is introduced as optional (could be enabled by variable
DEBCACHE_ENABLED).
2. Support for binary cache importing, the feature becomes official and
dpkg installation way is deprecated.
Alexander Smirnov (2):
classes/dpkg: Split install for cache
doc/technical_overview: Describe binary cache
Baurzhan Ismagulov (1):
classes/image: Provide /dev/null for Stretch apt
Frank Lenormand (3):
meta-isar-bin: Enable caching of deb packages
meta-isar-bin: Enable apt repo generation for amd64
classes/dpkg: Properly update packages in the cache
doc/technical_overview.md | 59 ++++++-
meta-isar-bin/conf/layer.conf | 18 +++
meta-isar-bin/files/distributions.in | 3 +
meta-isar/conf/bblayers.conf.sample | 1 +
.../images/files/debian-configscript.sh | 8 +
.../images/files/raspbian-configscript.sh | 8 +
meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
meta/classes/dpkg.bbclass | 174 ++++++++++++++++++++-
meta/classes/image.bbclass | 41 +++--
.../buildchroot/files/configscript.sh | 1 +
10 files changed, 297 insertions(+), 24 deletions(-)
create mode 100644 meta-isar-bin/conf/layer.conf
create mode 100644 meta-isar-bin/files/distributions.in
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
@ 2017-08-27 15:13 ` Alexander Smirnov
2017-08-28 15:18 ` Henning Schild
2017-09-06 14:21 ` Henning Schild
2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov
` (4 subsequent siblings)
5 siblings, 2 replies; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Frank Lenormand, Alexander Smirnov
From: Frank Lenormand <lenormf@gmail.com>
Adds the possibility to create an apt repository for newly built packages
and then use that repo for populating the target filesystem.
Signed-off-by: Frank Lenormand <lenormf@gmail.com>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar-bin/conf/layer.conf | 18 +++
meta-isar-bin/files/distributions.in | 3 +
meta-isar/conf/bblayers.conf.sample | 1 +
.../images/files/debian-configscript.sh | 8 ++
.../images/files/raspbian-configscript.sh | 8 ++
meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
meta/classes/dpkg.bbclass | 126 +++++++++++++++++++--
meta/classes/image.bbclass | 30 +++--
8 files changed, 186 insertions(+), 16 deletions(-)
create mode 100644 meta-isar-bin/conf/layer.conf
create mode 100644 meta-isar-bin/files/distributions.in
diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
new file mode 100644
index 0000000..24534e1
--- /dev/null
+++ b/meta-isar-bin/conf/layer.conf
@@ -0,0 +1,18 @@
+# Enable package caching with '1'
+DEBCACHE_ENABLED ?= "0"
+
+# Codename of the repository created by the caching class
+DEBDISTRONAME = "isar"
+
+# Path to the caching repository
+DEBCACHEDIR ?= "${LAYERDIR}/apt"
+
+# Path to the mount point of the repository within the target rootfs, during
+# population
+DEBCACHEMNT ?= "/opt/cache/apt"
+
+# Path to the databases used by `reprepro`
+DEBDBDIR ?= "${LAYERDIR}/db"
+
+# Path to the configuration files templates used by `reprepro`
+DEBFILESDIR ?= "${LAYERDIR}/files"
diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in
new file mode 100644
index 0000000..59f4429
--- /dev/null
+++ b/meta-isar-bin/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf source
+Components: main
diff --git a/meta-isar/conf/bblayers.conf.sample b/meta-isar/conf/bblayers.conf.sample
index 80867e7..53a362b 100644
--- a/meta-isar/conf/bblayers.conf.sample
+++ b/meta-isar/conf/bblayers.conf.sample
@@ -8,6 +8,7 @@ BBFILES ?= ""
BBLAYERS ?= " \
##ISARROOT##/meta \
##ISARROOT##/meta-isar \
+ ##ISARROOT##/meta-isar-bin \
"
BBLAYERS_NON_REMOVABLE ?= " \
##ISARROOT##/meta \
diff --git a/meta-isar/recipes-core/images/files/debian-configscript.sh b/meta-isar/recipes-core/images/files/debian-configscript.sh
index 4ac37d0..b05babb 100644
--- a/meta-isar/recipes-core/images/files/debian-configscript.sh
+++ b/meta-isar/recipes-core/images/files/debian-configscript.sh
@@ -8,6 +8,8 @@ set -e
readonly MACHINE_SERIAL="$1"
readonly BAUDRATE_TTY="$2"
readonly ROOTFS_DEV="$3"
+readonly DEBCACHEMNT="$4"
+readonly DEBDISTRONAME="$5"
cat >> /etc/default/locale << EOF
LANG=en_US.UTF-8
@@ -83,3 +85,9 @@ fi
if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ]; then
rm -f $TARGET/usr/sbin/policy-rc.d
fi
+
+mkdir -p /etc/apt/sources.list.d/
+cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
+deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
+deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
+EOF
diff --git a/meta-isar/recipes-core/images/files/raspbian-configscript.sh b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
index 2454481..f995f4d 100644
--- a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
+++ b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
@@ -8,6 +8,8 @@ set -e
readonly MACHINE_SERIAL="$1"
readonly BAUDRATE_TTY="$2"
readonly ROOTFS_DEV="$3"
+readonly DEBCACHEMNT="$4"
+readonly DEBDISTRONAME="$5"
cat >> /etc/default/locale << EOF
LANG=en_US.UTF-8
@@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
cat > /boot/config.txt << EOF
kernel=$KERNEL_IMAGE
EOF
+
+mkdir -p /etc/apt/sources.list.d/
+cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
+deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
+deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
+EOF
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index b679d97..85e6b27 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -49,8 +49,12 @@ do_rootfs() {
sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f "${WORKDIR}/multistrap.conf" || true
# Configure root filesystem
- sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL} ${BAUDRATE_TTY} \
- ${ROOTFS_DEV}
+ sudo chroot ${S} /configscript.sh \
+ ${MACHINE_SERIAL} \
+ ${BAUDRATE_TTY} \
+ ${ROOTFS_DEV} \
+ ${DEBCACHEMNT} \
+ ${DEBDISTRONAME}
sudo rm ${S}/configscript.sh
}
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 360a95c..b1e201d 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -1,5 +1,5 @@
# This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
+# Copyright (C) 2015-2017 ilbers GmbH
# Add dependency from buildchroot creation
DEPENDS += "buildchroot"
@@ -55,12 +55,124 @@ do_build() {
sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
}
-
# Install package to dedicated deploy directory
-do_install() {
- install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
+do_binary_deb_install() {
+ readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
+ readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
+
+ if [ "${DEBCACHE_ENABLED}" != "0" ]; then
+ # If `bitbake` is running for the first time, the cache doesn't exist
+ # yet and needs to be configured using a `distributions` file.
+ # A template stored in the layer directory is pre-processed to
+ # generate the configuration file, which is then placed in the
+ # appropriate directory.
+ if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
+ mkdir -p "${DIR_CACHE}/conf"
+ sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
+ "${DEBFILESDIR}/distributions.in" \
+ > "${DIR_CACHE}/conf/distributions"
+ fi
+
+ # Add binary and source packages to the deb cache
+ # If the cache doesn't exist yet, it will be created using the
+ # `distributions` file generated above.
+ ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do
+ reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
+ -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
+ done
+ else
+ # Deb caching is disabled, simply copy all binary packages to the
+ # deploy directory
+ mkdir -p "${DEPLOY_DIR_DEB}"
+ install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
+ fi
}
-addtask install after do_build
-do_install[dirs] = "${DEPLOY_DIR_DEB}"
-do_install[stamp-extra-info] = "${MACHINE}"
+addtask binary_deb_install after do_build
+do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
+# Deb caching lambda run during the parsing phase that checks whether the
+# current package has to be rebuilt, or taken from the cache
+python __anonymous () {
+ if d.getVar("DEBCACHE_ENABLED", True) == "0":
+ # Deb caching is disabled, do nothing
+ return True
+
+ PN = d.getVar("PN", True)
+ PV = d.getVar("PV", True)
+ DISTRO_ARCH = d.getVar("DISTRO_ARCH", True)
+ DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
+ DEBDISTRONAME = d.getVar("DEBDISTRONAME", True)
+ DEBDBDIR = d.getVar("DEBDBDIR", True)
+ DISTRO = d.getVar("DISTRO", True)
+ path_cache = os.path.join(DEBCACHEDIR, DISTRO)
+ path_databases = os.path.join(DEBDBDIR, DISTRO)
+ path_distributions = os.path.join(path_cache, "conf", "distributions")
+
+ # The distributions file is needed by `reprepro` to know what types
+ # of packages are supported, what the distribution name is, etc.
+ # If it doesn't exist, we have nothing in the cache, do nothing.
+ if not os.path.exists(path_distributions):
+ return
+
+ # Anonymous functions are run several times under different contexts
+ # during the parsing phase, which would let the code that follows be run
+ # as many times for the same package.
+ # In order to guarantee that our subroutine only runs once per package, we
+ # use bitbake's "persist" API in order to have reliable persistent storage
+ # accross calls of the lambda (using a simple variable in the class won't
+ # work, as several contexts won't allow fetching its value).
+ pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d)
+ if PN in pd and pd[PN] == PV:
+ return
+
+ import subprocess
+ try:
+ # The databases used by `reprepro` are not stored within the cache in
+ # order to make versioning of only the files needed to use the cache
+ # as an official Debian repository simpler.
+ # As such, if a developer uses a peer's cache to speed up their build
+ # time but have never run bitbake, the database will not have been
+ # created, so we regenerate them here.
+ if not os.path.exists(path_databases) and os.path.exists(path_cache):
+ bb.note("Regenerating the cache databases...")
+ subprocess.check_call([
+ "reprepro",
+ "--waitforlock", "3",
+ "-b", path_cache,
+ "--dbdir", path_databases,
+ "export", DEBDISTRONAME,
+ ])
+
+ # Get a list of the versions of all the packages named after the
+ # current bitbake package, and check whether the current package
+ # version is returned.
+ # As `reprepro` always returns zero with this particular operation, we
+ # have to use this workaround to check for a package in the cache.
+ package_version = subprocess.check_output([
+ "reprepro",
+ "--waitforlock", "3",
+ "-b", path_cache,
+ "--dbdir", path_databases,
+ "-C", "main",
+ "-A", DISTRO_ARCH,
+ "--list-format", "${version}",
+ "list", DEBDISTRONAME, PN,
+ ])
+ package_version = package_version.decode("utf-8")
+ if package_version == PV:
+ # The below list contains the names of all the tasks are in charge
+ # of building the package when the cache isn't enabled or if the
+ # package hasn't been placed in it already.
+ # As all tasks are enabled by default, we prevent their execution
+ # by setting the `noexec` flag, which will prevent a rebuild of
+ # the package when it's cached.
+ for task in ["fetch", "unpack", "build", "install"]:
+ d.setVarFlag("do_{}".format(task), "noexec", "1")
+
+ # Cache the results of this command so that subsequent executions
+ # of this anonymous functions don't run the same code again.
+ pd[PN] = PV
+ except subprocess.CalledProcessError as e:
+ bb.fatal("Unable to check for a candidate for package {0} (errorcode: {1})".format(PN, e.returncode))
+}
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a7f0d74..f42aa48 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
-# Install Debian packages, that were built from sources
+# Install Debian packages from the cache
do_populate() {
+ readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
+
if [ -n "${IMAGE_INSTALL}" ]; then
- sudo mkdir -p ${S}/deb
+ if [ "${DEBCACHE_ENABLED}" != "0" ]; then
+ sudo mkdir -p "${S}/${DEBCACHEMNT}"
+ sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
+
+ sudo chroot "${S}" apt-get update -y
+ for package in ${IMAGE_INSTALL}; do
+ sudo chroot "${S}" apt-get install -t "${DEBDISTRONAME}" -y \
+ --allow-unauthenticated "${package}"
+ done
+
+ sudo umount "${S}/${DEBCACHEMNT}"
+ else
+ sudo mkdir -p ${S}/deb
- for p in ${IMAGE_INSTALL}; do
- sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
- done
+ for p in ${IMAGE_INSTALL}; do
+ find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec \
+ sudo cp '{}' "${S}/deb/" \;
+ done
- sudo chroot ${S} /usr/bin/dpkg -i -R /deb
+ sudo chroot ${S} /usr/bin/dpkg -i -R /deb
- sudo rm -rf ${S}/deb
+ sudo rm -rf ${S}/deb
+ fi
fi
}
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
@ 2017-08-27 15:13 ` Alexander Smirnov
2017-08-28 15:20 ` Henning Schild
2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov
` (3 subsequent siblings)
5 siblings, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Baurzhan Ismagulov, Alexander Smirnov
From: Baurzhan Ismagulov <ibr@ilbers.de>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de>
---
meta/classes/image.bbclass | 3 +++
meta/recipes-devtools/buildchroot/files/configscript.sh | 1 +
2 files changed, 4 insertions(+)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index f42aa48..c2ff453 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -20,11 +20,14 @@ do_populate() {
sudo mkdir -p "${S}/${DEBCACHEMNT}"
sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
+ # apt-get http method, gpg require /dev/null
+ # Otherwise, apt http method hangs on Stretch
sudo chroot "${S}" apt-get update -y
for package in ${IMAGE_INSTALL}; do
sudo chroot "${S}" apt-get install -t "${DEBDISTRONAME}" -y \
--allow-unauthenticated "${package}"
done
+ sudo umount "${S}/dev"
sudo umount "${S}/${DEBCACHEMNT}"
else
diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh b/meta/recipes-devtools/buildchroot/files/configscript.sh
index 5080ae1..08c818b 100644
--- a/meta/recipes-devtools/buildchroot/files/configscript.sh
+++ b/meta/recipes-devtools/buildchroot/files/configscript.sh
@@ -40,6 +40,7 @@ export LC_ALL=C LANGUAGE=C LANG=C
/var/lib/dpkg/info/dash.preinst install
# apt-get http method, gpg require /dev/null
+# Otherwise, apt http method hangs on Stretch
mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev
#configuring packages
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 3/6] classes/dpkg: Split install for cache
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov
@ 2017-08-27 15:13 ` Alexander Smirnov
2017-08-28 8:00 ` Claudius Heine
2017-08-28 15:30 ` Henning Schild
2017-08-27 15:13 ` [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 Alexander Smirnov
` (2 subsequent siblings)
5 siblings, 2 replies; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Split install function into two parts:
- install_to_cache: if caching is enabled
- install_to_deploy: if caching is disabled
This patch brings flexibility to the implementation and makes it possible
to move all the caching implementation code to dedicated class. The magic
behavior depending on the value of DEBCACHE_ENABLED is now transparent in
the bitbake pipeline (can be inspected via "bitbake -g").
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/dpkg.bbclass | 66 ++++++++++++++++++++++++++--------------------
meta/classes/image.bbclass | 2 +-
2 files changed, 38 insertions(+), 30 deletions(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index b1e201d..118ba2f 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -56,46 +56,53 @@ do_build() {
}
# Install package to dedicated deploy directory
-do_binary_deb_install() {
+do_install_to_deploy() {
+ # Deb caching is disabled, simply copy all binary packages to the deploy
+ # directory
+ install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
+}
+
+addtask install_to_deploy after do_build
+do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}"
+do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_install_to_deploy[noexec] = "1"
+
+# Install package to dedicated apt repo
+do_install_to_cache() {
readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
- if [ "${DEBCACHE_ENABLED}" != "0" ]; then
- # If `bitbake` is running for the first time, the cache doesn't exist
- # yet and needs to be configured using a `distributions` file.
- # A template stored in the layer directory is pre-processed to
- # generate the configuration file, which is then placed in the
- # appropriate directory.
- if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
- mkdir -p "${DIR_CACHE}/conf"
- sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
- "${DEBFILESDIR}/distributions.in" \
- > "${DIR_CACHE}/conf/distributions"
- fi
-
- # Add binary and source packages to the deb cache
- # If the cache doesn't exist yet, it will be created using the
- # `distributions` file generated above.
- ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do
- reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
- -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
- done
- else
- # Deb caching is disabled, simply copy all binary packages to the
- # deploy directory
- mkdir -p "${DEPLOY_DIR_DEB}"
- install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
+ # If `bitbake` is running for the first time, the cache doesn't exist
+ # yet and needs to be configured using a `distributions` file.
+ # A template stored in the layer directory is pre-processed to
+ # generate the configuration file, which is then placed in the
+ # appropriate directory.
+ if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
+ mkdir -p "${DIR_CACHE}/conf"
+ sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
+ "${DEBFILESDIR}/distributions.in" \
+ > "${DIR_CACHE}/conf/distributions"
fi
+
+ # Add binary and source packages to the deb cache
+ # If the cache doesn't exist yet, it will be created using the
+ # `distributions` file generated above.
+ ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do
+ reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
+ -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
+ done
}
-addtask binary_deb_install after do_build
-do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+addtask install_to_cache after do_build
+do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+do_install_to_cache[noexec] = "1"
# Deb caching lambda run during the parsing phase that checks whether the
# current package has to be rebuilt, or taken from the cache
python __anonymous () {
if d.getVar("DEBCACHE_ENABLED", True) == "0":
- # Deb caching is disabled, do nothing
+ # Deb caching is disabled
+ d.delVarFlag('do_install_to_deploy', 'noexec')
return True
PN = d.getVar("PN", True)
@@ -108,6 +115,7 @@ python __anonymous () {
path_cache = os.path.join(DEBCACHEDIR, DISTRO)
path_databases = os.path.join(DEBDBDIR, DISTRO)
path_distributions = os.path.join(path_cache, "conf", "distributions")
+ d.delVarFlag('do_install_to_cache', 'noexec')
# The distributions file is needed by `reprepro` to know what types
# of packages are supported, what the distribution name is, etc.
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index c2ff453..6b1b5eb 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -46,4 +46,4 @@ do_populate() {
}
addtask populate before do_build
-do_populate[deptask] = "do_install"
+do_populate[deptask] = "do_install_to_cache do_install_to_deploy"
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
` (2 preceding siblings ...)
2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov
@ 2017-08-27 15:13 ` Alexander Smirnov
2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov
2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov
5 siblings, 0 replies; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Frank Lenormand, Alexander Smirnov
From: Frank Lenormand <lenormf@gmail.com>
Add amd64 architecture to binary caching.
Signed-off-by: Frank Lenormand <lenormf@gmail.com>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar-bin/files/distributions.in | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in
index 59f4429..cd214c6 100644
--- a/meta-isar-bin/files/distributions.in
+++ b/meta-isar-bin/files/distributions.in
@@ -1,3 +1,3 @@
Codename: {DISTRO_NAME}
-Architectures: i386 armhf source
+Architectures: i386 armhf amd64 source
Components: main
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 5/6] classes/dpkg: Properly update packages in the cache
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
` (3 preceding siblings ...)
2017-08-27 15:13 ` [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 Alexander Smirnov
@ 2017-08-27 15:13 ` Alexander Smirnov
2017-08-28 15:32 ` Henning Schild
2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov
5 siblings, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Frank Lenormand, Alexander Smirnov
From: Frank Lenormand <lenormf@gmail.com>
Before installing to cache, existing packages with the same name and
version have to be removed.
Signed-off-by: Frank Lenormand <lenormf@gmail.com>
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/dpkg.bbclass | 46 +++++++++++++++++++++++++++++++++++++++++++---
1 file changed, 43 insertions(+), 3 deletions(-)
diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
index 118ba2f..72ae79f 100644
--- a/meta/classes/dpkg.bbclass
+++ b/meta/classes/dpkg.bbclass
@@ -84,12 +84,52 @@ do_install_to_cache() {
> "${DIR_CACHE}/conf/distributions"
fi
+ print_field_value() {
+ awk "\$1 == \"${1}:\" { print \$2; }"
+ }
+
+ call_reprepro() {
+ reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
+ -C main "$@"
+ }
+
# Add binary and source packages to the deb cache
# If the cache doesn't exist yet, it will be created using the
# `distributions` file generated above.
- ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do
- reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
- -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
+ ls -1 "${BUILDROOT}"/*.deb | while read -r p; do
+ name_package=$(dpkg -f "${p}" | print_field_value "Package")
+ version_package=$(dpkg -f "${p}" | print_field_value "Version")
+
+ # Remove all packages with the same version that were added to the
+ # repository in previous builds
+ call_reprepro \
+ -A "${DISTRO_ARCH}" \
+ removefilter "${DEBDISTRONAME}" \
+ "Package (== ${name_package}), \
+ Version (== ${version_package})"
+ call_reprepro \
+ -A "${DISTRO_ARCH}" \
+ "include${p##*.}" \
+ "${DEBDISTRONAME}" \
+ "${p}"
+ done
+
+ ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do
+ name_package=$(cat "${p}" | print_field_value "Source")
+ version_package=$(cat "${p}" | print_field_value "Version")
+
+ # Remove all source packages with the same version that were added to
+ # the repository in previous builds
+ call_reprepro \
+ -A "source" \
+ removefilter "${DEBDISTRONAME}" \
+ "Package (== ${name_package}), \
+ Version (== ${version_package})"
+ call_reprepro \
+ -A "source" \
+ "include${p##*.}" \
+ "${DEBDISTRONAME}" \
+ "${p}"
done
}
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* [PATCH 6/6] doc/technical_overview: Describe binary cache
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
` (4 preceding siblings ...)
2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov
@ 2017-08-27 15:13 ` Alexander Smirnov
2017-08-28 15:36 ` Henning Schild
5 siblings, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-27 15:13 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Describe Isar binary cache feature.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
doc/technical_overview.md | 59 +++++++++++++++++++++++++++++++++++++++++++----
1 file changed, 55 insertions(+), 4 deletions(-)
diff --git a/doc/technical_overview.md b/doc/technical_overview.md
index 20e08b4..b3265dc 100644
--- a/doc/technical_overview.md
+++ b/doc/technical_overview.md
@@ -100,6 +100,43 @@ Target filesystem lifecycle can be described as following:
- According to the list of custom packages in bitbake recipes, the initial
filesystem will be populated by successfully built packages.
+## 2.5 Binary Cache
+
+As mentioned above, apart from fetching binary packages from upstream apt
+repositories, Isar is able to build custom packages from sources. To simplify
+further custom packages management, Isar implements binary caching.
+
+Binary cache is a typical apt repository created by Isar to store newly built
+packages. This repository has mainly two goals:
+
+ - Simplify image filesystem populating and distributing binary packages to
+ other systems.
+
+ - Reuse these binaries in next builds to avoid re-building from the
+ scratch everytime.
+
+Current Isar tree provides default cache repository: `meta-isar-bin`.
+Isar binary cache is described by the following files:
+
+ - `meta-isar-bin/conf/layer.conf`
+
+ - `meta-isar-bin/files/distributions.in`
+
+After system build is complete, this repo contains binary packages for all the
+distros and architectures, requested in multiconfig. So this folder can be
+published and used as apt repo for already installed systems.
+
+An important note, that Isar creates local repositories using dedicated Debian
+tool reprepro. This tool takes care about:
+
+ - Maintenance of multiple architectures and distros.
+
+ - Internal packages database management.
+
+ - Repo signatures management.
+
+ - Requests about available packages in repo (name, version etc...).
+
# 3 Isar Internal Processes
## 3.1 General Overview
@@ -174,10 +211,17 @@ contain debian folder. The build process is implemented in
3. Run dpkg-buildpackage
-4. Task `do_install`: install successfully built packages
+4. Further package processing depends on binary cache option:
+
+ 1. Task `do_install_to_deploy`: install successfully built packages if
+ binary cache is disabled:
`${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
`${DEPLOY_DIR_DEB}`
+ 2. Task `do_install_to_cache`: install successfully built packages to
+ binary cache repository:
+ `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}`
+
## 3.5 Populate Target Filesystem
Each target image can be extended by custom packages listed in IMAGE_INSTALL
@@ -185,11 +229,18 @@ variable. Task `do_populate` performs the following:
1. Parse IMAGE_INSTALL variable.
-2. Find respective packages in `${DEPLOY_DIR_DEB}`.
+2. Further steps depends in binary cache option.
+If binary cache is disabled:
+
+ 1. Find respective packages in `${DEPLOY_DIR_DEB}`.
+
+ 2. Copy packages to deb folder in dedicated target filesystem.
+
+ 3. Execute dpkg command in chroot for all the copied packages.
-3. Copy them to deb folder in dedicated target filesystem.
+If binary cache is enabled:
-4. Execute dpkg command in chroot for all the copied packages.
+ 1. Call apt-get install in chroot in target image filesystem.
## 3.6 Generate Bootable Image
--
2.1.4
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache
2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov
@ 2017-08-28 8:00 ` Claudius Heine
2017-08-29 7:18 ` Alexander Smirnov
2017-08-28 15:30 ` Henning Schild
1 sibling, 1 reply; 30+ messages in thread
From: Claudius Heine @ 2017-08-28 8:00 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
Hi,
On 08/27/2017 05:13 PM, Alexander Smirnov wrote:
> Split install function into two parts:
> - install_to_cache: if caching is enabled
> - install_to_deploy: if caching is disabled
>
> This patch brings flexibility to the implementation and makes it possible
> to move all the caching implementation code to dedicated class. The magic
> behavior depending on the value of DEBCACHE_ENABLED is now transparent in
> the bitbake pipeline (can be inspected via "bitbake -g").
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/dpkg.bbclass | 66 ++++++++++++++++++++++++++--------------------
> meta/classes/image.bbclass | 2 +-
> 2 files changed, 38 insertions(+), 30 deletions(-)
>
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index b1e201d..118ba2f 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -56,46 +56,53 @@ do_build() {
> }
>
> # Install package to dedicated deploy directory
> -do_binary_deb_install() {
> +do_install_to_deploy() {
> + # Deb caching is disabled, simply copy all binary packages to the deploy
> + # directory
> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> +}
> +
> +addtask install_to_deploy after do_build
> +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}"
> +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_install_to_deploy[noexec] = "1"
> +
> +# Install package to dedicated apt repo
> +do_install_to_cache() {
> readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
>
> - if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> - # If `bitbake` is running for the first time, the cache doesn't exist
> - # yet and needs to be configured using a `distributions` file.
> - # A template stored in the layer directory is pre-processed to
> - # generate the configuration file, which is then placed in the
> - # appropriate directory.
> - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> - mkdir -p "${DIR_CACHE}/conf"
> - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> - "${DEBFILESDIR}/distributions.in" \
> - > "${DIR_CACHE}/conf/distributions"
> - fi
> -
> - # Add binary and source packages to the deb cache
> - # If the cache doesn't exist yet, it will be created using the
> - # `distributions` file generated above.
> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do
> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> - done
> - else
> - # Deb caching is disabled, simply copy all binary packages to the
> - # deploy directory
> - mkdir -p "${DEPLOY_DIR_DEB}"
> - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> + # If `bitbake` is running for the first time, the cache doesn't exist
> + # yet and needs to be configured using a `distributions` file.
> + # A template stored in the layer directory is pre-processed to
> + # generate the configuration file, which is then placed in the
> + # appropriate directory.
> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> + mkdir -p "${DIR_CACHE}/conf"
> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> + "${DEBFILESDIR}/distributions.in" \
> + > "${DIR_CACHE}/conf/distributions"
> fi
> +
> + # Add binary and source packages to the deb cache
> + # If the cache doesn't exist yet, it will be created using the
> + # `distributions` file generated above.
> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r p; do
> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> + done
> }
>
> -addtask binary_deb_install after do_build
> -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +addtask install_to_cache after do_build
What was the reason why we put stuff after the 'build' task? In OE the
'build' task is the default task triggering everything necessary for a
recipe. In Isar that changed. Why?
> +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_install_to_cache[noexec] = "1"
>
> # Deb caching lambda run during the parsing phase that checks whether the
> # current package has to be rebuilt, or taken from the cache
> python __anonymous () {
> if d.getVar("DEBCACHE_ENABLED", True) == "0":
> - # Deb caching is disabled, do nothing
> + # Deb caching is disabled
> + d.delVarFlag('do_install_to_deploy', 'noexec')
> return True
>
> PN = d.getVar("PN", True)
> @@ -108,6 +115,7 @@ python __anonymous () {
> path_cache = os.path.join(DEBCACHEDIR, DISTRO)
> path_databases = os.path.join(DEBDBDIR, DISTRO)
> path_distributions = os.path.join(path_cache, "conf", "distributions")
> + d.delVarFlag('do_install_to_cache', 'noexec')
>
> # The distributions file is needed by `reprepro` to know what types
> # of packages are supported, what the distribution name is, etc.
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c2ff453..6b1b5eb 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -46,4 +46,4 @@ do_populate() {
> }
>
> addtask populate before do_build
> -do_populate[deptask] = "do_install"
> +do_populate[deptask] = "do_install_to_cache do_install_to_deploy"
I would rather still have a, maybe empty, 'do_install' step that just
depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks. I
don't see why ever small change in one class should change the complete
pipeline for every recipe.
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
@ 2017-08-28 15:18 ` Henning Schild
2017-08-29 6:40 ` Alexander Smirnov
2017-08-31 10:55 ` Claudius Heine
2017-09-06 14:21 ` Henning Schild
1 sibling, 2 replies; 30+ messages in thread
From: Henning Schild @ 2017-08-28 15:18 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users, Frank Lenormand
This patch is very big, doing a lot of things at a time.
The commit message on the other hand is not very verbose.
creating the repo is one step
- i think using reprepro is a good idea
moving do_populate to that repo is another one
- i think this task should go away and the repo should be given to
multistrap
and there are probably more ways to split that into reviewable pieces
So this review is not too verbose, we need smaller pieces.
More inline
Am Sun, 27 Aug 2017 18:13:34 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> From: Frank Lenormand <lenormf@gmail.com>
>
> Adds the possibility to create an apt repository for newly built
> packages and then use that repo for populating the target filesystem.
>
> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar-bin/conf/layer.conf | 18 +++
> meta-isar-bin/files/distributions.in | 3 +
> meta-isar/conf/bblayers.conf.sample | 1 +
> .../images/files/debian-configscript.sh | 8 ++
> .../images/files/raspbian-configscript.sh | 8 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
> meta/classes/dpkg.bbclass | 126
> +++++++++++++++++++--
> meta/classes/image.bbclass | 30 +++-- 8
> files changed, 186 insertions(+), 16 deletions(-) create mode 100644
> meta-isar-bin/conf/layer.conf create mode 100644
> meta-isar-bin/files/distributions.in
>
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf new file mode 100644
> index 0000000..24534e1
> --- /dev/null
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -0,0 +1,18 @@
> +# Enable package caching with '1'
> +DEBCACHE_ENABLED ?= "0"
> +
> +# Codename of the repository created by the caching class
> +DEBDISTRONAME = "isar"
> +
> +# Path to the caching repository
> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> +
> +# Path to the mount point of the repository within the target
> rootfs, during +# population
> +DEBCACHEMNT ?= "/opt/cache/apt"
> +
> +# Path to the databases used by `reprepro`
> +DEBDBDIR ?= "${LAYERDIR}/db"
> +
> +# Path to the configuration files templates used by `reprepro`
> +DEBFILESDIR ?= "${LAYERDIR}/files"
> diff --git a/meta-isar-bin/files/distributions.in
> b/meta-isar-bin/files/distributions.in new file mode 100644
> index 0000000..59f4429
> --- /dev/null
> +++ b/meta-isar-bin/files/distributions.in
> @@ -0,0 +1,3 @@
> +Codename: {DISTRO_NAME}
> +Architectures: i386 armhf source
> +Components: main
> diff --git a/meta-isar/conf/bblayers.conf.sample
> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> --- a/meta-isar/conf/bblayers.conf.sample
> +++ b/meta-isar/conf/bblayers.conf.sample
> @@ -8,6 +8,7 @@ BBFILES ?= ""
> BBLAYERS ?= " \
> ##ISARROOT##/meta \
> ##ISARROOT##/meta-isar \
> + ##ISARROOT##/meta-isar-bin \
> "
> BBLAYERS_NON_REMOVABLE ?= " \
> ##ISARROOT##/meta \
> diff --git
> a/meta-isar/recipes-core/images/files/debian-configscript.sh
> b/meta-isar/recipes-core/images/files/debian-configscript.sh index
> 4ac37d0..b05babb 100644 ---
> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++
> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6
> +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2"
> readonly ROOTFS_DEV="$3"
> +readonly DEBCACHEMNT="$4"
> +readonly DEBDISTRONAME="$5"
>
> cat >> /etc/default/locale << EOF
> LANG=en_US.UTF-8
> @@ -83,3 +85,9 @@ fi
> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ];
> then rm -f $TARGET/usr/sbin/policy-rc.d
> fi
> +
> +mkdir -p /etc/apt/sources.list.d/
> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +EOF
> diff --git
> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index
> 2454481..f995f4d 100644 ---
> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++
> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@
> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
> +readonly DEBCACHEMNT="$4"
> +readonly DEBDISTRONAME="$5"
>
> cat >> /etc/default/locale << EOF
> LANG=en_US.UTF-8
> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
> cat > /boot/config.txt << EOF
> kernel=$KERNEL_IMAGE
> EOF
> +
> +mkdir -p /etc/apt/sources.list.d/
> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +EOF
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> b679d97..85e6b27 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@
> do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true
> # Configure root filesystem
> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> ${BAUDRATE_TTY} \
> - ${ROOTFS_DEV}
> + sudo chroot ${S} /configscript.sh \
> + ${MACHINE_SERIAL} \
> + ${BAUDRATE_TTY} \
> + ${ROOTFS_DEV} \
> + ${DEBCACHEMNT} \
> + ${DEBDISTRONAME}
> sudo rm ${S}/configscript.sh
> }
>
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 360a95c..b1e201d 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -1,5 +1,5 @@
> # This software is a part of ISAR.
> -# Copyright (C) 2015-2016 ilbers GmbH
> +# Copyright (C) 2015-2017 ilbers GmbH
>
> # Add dependency from buildchroot creation
> DEPENDS += "buildchroot"
> @@ -55,12 +55,124 @@ do_build() {
> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
> }
>
> -
> # Install package to dedicated deploy directory
> -do_install() {
> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
> +do_binary_deb_install() {
> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
> +
> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> + # If `bitbake` is running for the first time, the cache
> doesn't exist
> + # yet and needs to be configured using a `distributions`
> file.
> + # A template stored in the layer directory is pre-processed
> to
> + # generate the configuration file, which is then placed in
> the
> + # appropriate directory.
> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> + mkdir -p "${DIR_CACHE}/conf"
> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> + "${DEBFILESDIR}/distributions.in" \
> + > "${DIR_CACHE}/conf/distributions"
> + fi
This step should be in the image and not in every package. I guess it
has a race with multiple packages running this init routine.
> + # Add binary and source packages to the deb cache
> + # If the cache doesn't exist yet, it will be created using
> the
> + # `distributions` file generated above.
> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
> -r p; do
> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> "${DIR_DB}" \
> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> + done
> + else
> + # Deb caching is disabled, simply copy all binary packages
> to the
> + # deploy directory
> + mkdir -p "${DEPLOY_DIR_DEB}"
> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> + fi
> }
>
> -addtask install after do_build
> -do_install[dirs] = "${DEPLOY_DIR_DEB}"
> -do_install[stamp-extra-info] = "${MACHINE}"
> +addtask binary_deb_install after do_build
> +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
Here a task got renamed. This should be another patch ... one that i
already tried to get in.
> +
> +# Deb caching lambda run during the parsing phase that checks
> whether the +# current package has to be rebuilt, or taken from the
> cache +python __anonymous () {
> + if d.getVar("DEBCACHE_ENABLED", True) == "0":
> + # Deb caching is disabled, do nothing
> + return True
> +
> + PN = d.getVar("PN", True)
> + PV = d.getVar("PV", True)
> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True)
> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True)
> + DEBDBDIR = d.getVar("DEBDBDIR", True)
> + DISTRO = d.getVar("DISTRO", True)
> + path_cache = os.path.join(DEBCACHEDIR, DISTRO)
> + path_databases = os.path.join(DEBDBDIR, DISTRO)
> + path_distributions = os.path.join(path_cache, "conf",
> "distributions") +
> + # The distributions file is needed by `reprepro` to know what
> types
> + # of packages are supported, what the distribution name is, etc.
> + # If it doesn't exist, we have nothing in the cache, do nothing.
> + if not os.path.exists(path_distributions):
> + return
> +
> + # Anonymous functions are run several times under different
> contexts
> + # during the parsing phase, which would let the code that
> follows be run
> + # as many times for the same package.
> + # In order to guarantee that our subroutine only runs once per
> package, we
> + # use bitbake's "persist" API in order to have reliable
> persistent storage
> + # accross calls of the lambda (using a simple variable in the
> class won't
> + # work, as several contexts won't allow fetching its value).
> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d)
> + if PN in pd and pd[PN] == PV:
> + return
> +
> + import subprocess
> + try:
> + # The databases used by `reprepro` are not stored within the
> cache in
> + # order to make versioning of only the files needed to use
> the cache
> + # as an official Debian repository simpler.
> + # As such, if a developer uses a peer's cache to speed up
> their build
> + # time but have never run bitbake, the database will not
> have been
> + # created, so we regenerate them here.
> + if not os.path.exists(path_databases) and
> os.path.exists(path_cache):
> + bb.note("Regenerating the cache databases...")
> + subprocess.check_call([
> + "reprepro",
> + "--waitforlock", "3",
> + "-b", path_cache,
> + "--dbdir", path_databases,
> + "export", DEBDISTRONAME,
> + ])
> +
> + # Get a list of the versions of all the packages named after
> the
> + # current bitbake package, and check whether the current
> package
> + # version is returned.
> + # As `reprepro` always returns zero with this particular
> operation, we
> + # have to use this workaround to check for a package in the
> cache.
> + package_version = subprocess.check_output([
> + "reprepro",
> + "--waitforlock", "3",
> + "-b", path_cache,
> + "--dbdir", path_databases,
> + "-C", "main",
> + "-A", DISTRO_ARCH,
> + "--list-format", "${version}",
> + "list", DEBDISTRONAME, PN,
> + ])
> + package_version = package_version.decode("utf-8")
> + if package_version == PV:
> + # The below list contains the names of all the tasks are
> in charge
> + # of building the package when the cache isn't enabled
> or if the
> + # package hasn't been placed in it already.
> + # As all tasks are enabled by default, we prevent their
> execution
> + # by setting the `noexec` flag, which will prevent a
> rebuild of
> + # the package when it's cached.
> + for task in ["fetch", "unpack", "build", "install"]:
> + d.setVarFlag("do_{}".format(task), "noexec", "1")
> +
> + # Cache the results of this command so that subsequent
> executions
> + # of this anonymous functions don't run the same code
> again.
> + pd[PN] = PV
> + except subprocess.CalledProcessError as e:
> + bb.fatal("Unable to check for a candidate for package {0}
> (errorcode: {1})".format(PN, e.returncode)) +}
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index a7f0d74..f42aa48 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
>
> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>
> -# Install Debian packages, that were built from sources
> +# Install Debian packages from the cache
> do_populate() {
> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> +
> if [ -n "${IMAGE_INSTALL}" ]; then
> - sudo mkdir -p ${S}/deb
> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> + sudo mkdir -p "${S}/${DEBCACHEMNT}"
> + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
> +
> + sudo chroot "${S}" apt-get update -y
This will fail if you are behind a proxy. It makes a known issue worse.
> + for package in ${IMAGE_INSTALL}; do
> + sudo chroot "${S}" apt-get install -t
> "${DEBDISTRONAME}" -y \
> + --allow-unauthenticated "${package}"
> + done
> +
> + sudo umount "${S}/${DEBCACHEMNT}"
A whole lot of sudo. I do not mind but am still wondering ...
Can we replace the bind-mount with a symlink? If anything between the
mount and umount fails and you need to run again ... will the mount
fail if the old one is still there?
> + else
> + sudo mkdir -p ${S}/deb
>
> - for p in ${IMAGE_INSTALL}; do
> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
> - done
> + for p in ${IMAGE_INSTALL}; do
> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec
> \
> + sudo cp '{}' "${S}/deb/" \;
> + done
>
> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb
> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb
>
> - sudo rm -rf ${S}/deb
> + sudo rm -rf ${S}/deb
> + fi
> fi
> }
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt
2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov
@ 2017-08-28 15:20 ` Henning Schild
2017-08-28 15:26 ` Henning Schild
0 siblings, 1 reply; 30+ messages in thread
From: Henning Schild @ 2017-08-28 15:20 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users, Baurzhan Ismagulov
Am Sun, 27 Aug 2017 18:13:35 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> From: Baurzhan Ismagulov <ibr@ilbers.de>
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de>
> ---
> meta/classes/image.bbclass | 3 +++
> meta/recipes-devtools/buildchroot/files/configscript.sh | 1 +
> 2 files changed, 4 insertions(+)
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index f42aa48..c2ff453 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -20,11 +20,14 @@ do_populate() {
> sudo mkdir -p "${S}/${DEBCACHEMNT}"
> sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
>
> + # apt-get http method, gpg require /dev/null
> + # Otherwise, apt http method hangs on Stretch
> sudo chroot "${S}" apt-get update -y
> for package in ${IMAGE_INSTALL}; do
> sudo chroot "${S}" apt-get install -t
> "${DEBDISTRONAME}" -y \ --allow-unauthenticated "${package}"
> done
> + sudo umount "${S}/dev"
I see a new umount and a comment, where is the new mount?
Henning
> sudo umount "${S}/${DEBCACHEMNT}"
> else
> diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh
> b/meta/recipes-devtools/buildchroot/files/configscript.sh index
> 5080ae1..08c818b 100644 ---
> a/meta/recipes-devtools/buildchroot/files/configscript.sh +++
> b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -40,6
> +40,7 @@ export LC_ALL=C LANGUAGE=C
> LANG=C /var/lib/dpkg/info/dash.preinst install
> # apt-get http method, gpg require /dev/null
> +# Otherwise, apt http method hangs on Stretch
> mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev
>
> #configuring packages
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt
2017-08-28 15:20 ` Henning Schild
@ 2017-08-28 15:26 ` Henning Schild
0 siblings, 0 replies; 30+ messages in thread
From: Henning Schild @ 2017-08-28 15:26 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users, Baurzhan Ismagulov
Am Mon, 28 Aug 2017 17:20:06 +0200
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:
> Am Sun, 27 Aug 2017 18:13:35 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
> > From: Baurzhan Ismagulov <ibr@ilbers.de>
> >
> > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> > Signed-off-by: Baurzhan Ismagulov <ibr@ilbers.de>
> > ---
> > meta/classes/image.bbclass | 3 +++
> > meta/recipes-devtools/buildchroot/files/configscript.sh | 1 +
> > 2 files changed, 4 insertions(+)
> >
> > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> > index f42aa48..c2ff453 100644
> > --- a/meta/classes/image.bbclass
> > +++ b/meta/classes/image.bbclass
> > @@ -20,11 +20,14 @@ do_populate() {
> > sudo mkdir -p "${S}/${DEBCACHEMNT}"
> > sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
> >
> > + # apt-get http method, gpg require /dev/null
> > + # Otherwise, apt http method hangs on Stretch
> > sudo chroot "${S}" apt-get update -y
> > for package in ${IMAGE_INSTALL}; do
> > sudo chroot "${S}" apt-get install -t
> > "${DEBDISTRONAME}" -y \ --allow-unauthenticated "${package}"
> > done
> > + sudo umount "${S}/dev"
>
> I see a new umount and a comment, where is the new mount?
>
> Henning
>
> > sudo umount "${S}/${DEBCACHEMNT}"
> > else
> > diff --git a/meta/recipes-devtools/buildchroot/files/configscript.sh
> > b/meta/recipes-devtools/buildchroot/files/configscript.sh index
> > 5080ae1..08c818b 100644 ---
> > a/meta/recipes-devtools/buildchroot/files/configscript.sh +++
> > b/meta/recipes-devtools/buildchroot/files/configscript.sh @@ -40,6
> > +40,7 @@ export LC_ALL=C LANGUAGE=C
> > LANG=C /var/lib/dpkg/info/dash.preinst install
> > # apt-get http method, gpg require /dev/null
> > +# Otherwise, apt http method hangs on Stretch
> > mount -t devtmpfs -o mode=0755,nosuid devtmpfs /dev
Found it ... and it is not new.
Can we not get the "mount; { stuff } umount;" into one file?
But we should first discuss whether we will drop do_populate
alltogether. See
"[PATCH 3/3] classes: image: remove populate and replace it with a
custom repo"
Henning
> > #configuring packages
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache
2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov
2017-08-28 8:00 ` Claudius Heine
@ 2017-08-28 15:30 ` Henning Schild
1 sibling, 0 replies; 30+ messages in thread
From: Henning Schild @ 2017-08-28 15:30 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
This is a fixup of 1/6. No need for another patch, just fix 1of6.
Henning
Am Sun, 27 Aug 2017 18:13:36 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> Split install function into two parts:
> - install_to_cache: if caching is enabled
> - install_to_deploy: if caching is disabled
>
> This patch brings flexibility to the implementation and makes it
> possible to move all the caching implementation code to dedicated
> class. The magic behavior depending on the value of DEBCACHE_ENABLED
> is now transparent in the bitbake pipeline (can be inspected via
> "bitbake -g").
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/dpkg.bbclass | 66
> ++++++++++++++++++++++++++--------------------
> meta/classes/image.bbclass | 2 +- 2 files changed, 38 insertions(+),
> 30 deletions(-)
>
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index b1e201d..118ba2f 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -56,46 +56,53 @@ do_build() {
> }
>
> # Install package to dedicated deploy directory
> -do_binary_deb_install() {
> +do_install_to_deploy() {
> + # Deb caching is disabled, simply copy all binary packages to
> the deploy
> + # directory
> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> +}
> +
> +addtask install_to_deploy after do_build
> +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}"
> +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_install_to_deploy[noexec] = "1"
> +
> +# Install package to dedicated apt repo
> +do_install_to_cache() {
> readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
>
> - if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> - # If `bitbake` is running for the first time, the cache
> doesn't exist
> - # yet and needs to be configured using a `distributions`
> file.
> - # A template stored in the layer directory is pre-processed
> to
> - # generate the configuration file, which is then placed in
> the
> - # appropriate directory.
> - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> - mkdir -p "${DIR_CACHE}/conf"
> - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> - "${DEBFILESDIR}/distributions.in" \
> - > "${DIR_CACHE}/conf/distributions"
> - fi
> -
> - # Add binary and source packages to the deb cache
> - # If the cache doesn't exist yet, it will be created using
> the
> - # `distributions` file generated above.
> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
> -r p; do
> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> "${DIR_DB}" \
> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> - done
> - else
> - # Deb caching is disabled, simply copy all binary packages
> to the
> - # deploy directory
> - mkdir -p "${DEPLOY_DIR_DEB}"
> - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> + # If `bitbake` is running for the first time, the cache doesn't
> exist
> + # yet and needs to be configured using a `distributions` file.
> + # A template stored in the layer directory is pre-processed to
> + # generate the configuration file, which is then placed in the
> + # appropriate directory.
> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> + mkdir -p "${DIR_CACHE}/conf"
> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> + "${DEBFILESDIR}/distributions.in" \
> + > "${DIR_CACHE}/conf/distributions"
> fi
> +
> + # Add binary and source packages to the deb cache
> + # If the cache doesn't exist yet, it will be created using the
> + # `distributions` file generated above.
> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r
> p; do
> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> "${DIR_DB}" \
> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> + done
> }
>
> -addtask binary_deb_install after do_build
> -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +addtask install_to_cache after do_build
> +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +do_install_to_cache[noexec] = "1"
>
> # Deb caching lambda run during the parsing phase that checks
> whether the # current package has to be rebuilt, or taken from the
> cache python __anonymous () {
> if d.getVar("DEBCACHE_ENABLED", True) == "0":
> - # Deb caching is disabled, do nothing
> + # Deb caching is disabled
> + d.delVarFlag('do_install_to_deploy', 'noexec')
> return True
>
> PN = d.getVar("PN", True)
> @@ -108,6 +115,7 @@ python __anonymous () {
> path_cache = os.path.join(DEBCACHEDIR, DISTRO)
> path_databases = os.path.join(DEBDBDIR, DISTRO)
> path_distributions = os.path.join(path_cache, "conf",
> "distributions")
> + d.delVarFlag('do_install_to_cache', 'noexec')
>
> # The distributions file is needed by `reprepro` to know what
> types # of packages are supported, what the distribution name is, etc.
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index c2ff453..6b1b5eb 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -46,4 +46,4 @@ do_populate() {
> }
>
> addtask populate before do_build
> -do_populate[deptask] = "do_install"
> +do_populate[deptask] = "do_install_to_cache do_install_to_deploy"
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache
2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov
@ 2017-08-28 15:32 ` Henning Schild
2017-08-29 7:20 ` Alexander Smirnov
0 siblings, 1 reply; 30+ messages in thread
From: Henning Schild @ 2017-08-28 15:32 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users, Frank Lenormand
This is another fixup of 1/6, fold it in.
Henning
Am Sun, 27 Aug 2017 18:13:38 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> From: Frank Lenormand <lenormf@gmail.com>
>
> Before installing to cache, existing packages with the same name and
> version have to be removed.
>
> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta/classes/dpkg.bbclass | 46
> +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43
> insertions(+), 3 deletions(-)
>
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 118ba2f..72ae79f 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -84,12 +84,52 @@ do_install_to_cache() {
> > "${DIR_CACHE}/conf/distributions"
> fi
>
> + print_field_value() {
> + awk "\$1 == \"${1}:\" { print \$2; }"
> + }
> +
> + call_reprepro() {
> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> "${DIR_DB}" \
> + -C main "$@"
> + }
> +
> # Add binary and source packages to the deb cache
> # If the cache doesn't exist yet, it will be created using the
> # `distributions` file generated above.
> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r
> p; do
> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> "${DIR_DB}" \
> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do
> + name_package=$(dpkg -f "${p}" | print_field_value "Package")
> + version_package=$(dpkg -f "${p}" | print_field_value
> "Version") +
> + # Remove all packages with the same version that were added
> to the
> + # repository in previous builds
> + call_reprepro \
> + -A "${DISTRO_ARCH}" \
> + removefilter "${DEBDISTRONAME}" \
> + "Package (== ${name_package}), \
> + Version (== ${version_package})"
> + call_reprepro \
> + -A "${DISTRO_ARCH}" \
> + "include${p##*.}" \
> + "${DEBDISTRONAME}" \
> + "${p}"
> + done
> +
> + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do
> + name_package=$(cat "${p}" | print_field_value "Source")
> + version_package=$(cat "${p}" | print_field_value "Version")
> +
> + # Remove all source packages with the same version that were
> added to
> + # the repository in previous builds
> + call_reprepro \
> + -A "source" \
> + removefilter "${DEBDISTRONAME}" \
> + "Package (== ${name_package}), \
> + Version (== ${version_package})"
> + call_reprepro \
> + -A "source" \
> + "include${p##*.}" \
> + "${DEBDISTRONAME}" \
> + "${p}"
> done
> }
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache
2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov
@ 2017-08-28 15:36 ` Henning Schild
2017-08-29 7:29 ` Alexander Smirnov
2017-08-29 11:29 ` Jan Kiszka
0 siblings, 2 replies; 30+ messages in thread
From: Henning Schild @ 2017-08-28 15:36 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
This should be done along the way in every single patch, not afterwards.
Henning
Am Sun, 27 Aug 2017 18:13:39 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> Describe Isar binary cache feature.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> doc/technical_overview.md | 59
> +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55
> insertions(+), 4 deletions(-)
>
> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> index 20e08b4..b3265dc 100644
> --- a/doc/technical_overview.md
> +++ b/doc/technical_overview.md
> @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described as
> following:
> - According to the list of custom packages in bitbake recipes, the
> initial filesystem will be populated by successfully built packages.
>
> +## 2.5 Binary Cache
> +
> +As mentioned above, apart from fetching binary packages from
> upstream apt +repositories, Isar is able to build custom packages
> from sources. To simplify +further custom packages management, Isar
> implements binary caching. +
> +Binary cache is a typical apt repository created by Isar to store
> newly built +packages. This repository has mainly two goals:
> +
> + - Simplify image filesystem populating and distributing binary
> packages to
> + other systems.
> +
> + - Reuse these binaries in next builds to avoid re-building from the
> + scratch everytime.
> +
> +Current Isar tree provides default cache repository: `meta-isar-bin`.
> +Isar binary cache is described by the following files:
> +
> + - `meta-isar-bin/conf/layer.conf`
> +
> + - `meta-isar-bin/files/distributions.in`
> +
> +After system build is complete, this repo contains binary packages
> for all the +distros and architectures, requested in multiconfig. So
> this folder can be +published and used as apt repo for already
> installed systems. +
> +An important note, that Isar creates local repositories using
> dedicated Debian +tool reprepro. This tool takes care about:
> +
> + - Maintenance of multiple architectures and distros.
> +
> + - Internal packages database management.
> +
> + - Repo signatures management.
> +
> + - Requests about available packages in repo (name, version etc...).
> +
> # 3 Isar Internal Processes
>
> ## 3.1 General Overview
> @@ -174,10 +211,17 @@ contain debian folder. The build process is
> implemented in
> 3. Run dpkg-buildpackage
>
> -4. Task `do_install`: install successfully built packages
> +4. Further package processing depends on binary cache option:
> +
> + 1. Task `do_install_to_deploy`: install successfully built
> packages if
> + binary cache is disabled:
> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
> `${DEPLOY_DIR_DEB}`
>
> + 2. Task `do_install_to_cache`: install successfully built
> packages to
> + binary cache repository:
> + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}`
> +
> ## 3.5 Populate Target Filesystem
>
> Each target image can be extended by custom packages listed in
> IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate`
> performs the following:
> 1. Parse IMAGE_INSTALL variable.
>
> -2. Find respective packages in `${DEPLOY_DIR_DEB}`.
> +2. Further steps depends in binary cache option.
> +If binary cache is disabled:
> +
> + 1. Find respective packages in `${DEPLOY_DIR_DEB}`.
> +
> + 2. Copy packages to deb folder in dedicated target filesystem.
> +
> + 3. Execute dpkg command in chroot for all the copied packages.
>
> -3. Copy them to deb folder in dedicated target filesystem.
> +If binary cache is enabled:
>
> -4. Execute dpkg command in chroot for all the copied packages.
> + 1. Call apt-get install in chroot in target image filesystem.
>
> ## 3.6 Generate Bootable Image
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-28 15:18 ` Henning Schild
@ 2017-08-29 6:40 ` Alexander Smirnov
2017-08-29 7:51 ` Henning Schild
2017-08-31 10:55 ` Claudius Heine
1 sibling, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-29 6:40 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 08/28/2017 06:18 PM, Henning Schild wrote:
> This patch is very big, doing a lot of things at a time.
Ok, I'll think how it can be split.
> The commit message on the other hand is not very verbose.
Half of the patch is the comments that describe in details what happens
in the code, so please questions and proposals.
>
> creating the repo is one step
> - i think using reprepro is a good idea
> moving do_populate to that repo is another one
> - i think this task should go away and the repo should be given to
> multistrap
Ok
> and there are probably more ways to split that into reviewable pieces
Please questions and proposals.
>
> So this review is not too verbose, we need smaller pieces.
>
> More inline
>
> Am Sun, 27 Aug 2017 18:13:34 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> From: Frank Lenormand <lenormf@gmail.com>
>>
>> Adds the possibility to create an apt repository for newly built
>> packages and then use that repo for populating the target filesystem.
>>
>> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta-isar-bin/conf/layer.conf | 18 +++
>> meta-isar-bin/files/distributions.in | 3 +
>> meta-isar/conf/bblayers.conf.sample | 1 +
>> .../images/files/debian-configscript.sh | 8 ++
>> .../images/files/raspbian-configscript.sh | 8 ++
>> meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
>> meta/classes/dpkg.bbclass | 126
>> +++++++++++++++++++--
>> meta/classes/image.bbclass | 30 +++-- 8
>> files changed, 186 insertions(+), 16 deletions(-) create mode 100644
>> meta-isar-bin/conf/layer.conf create mode 100644
>> meta-isar-bin/files/distributions.in
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf new file mode 100644
>> index 0000000..24534e1
>> --- /dev/null
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -0,0 +1,18 @@
>> +# Enable package caching with '1'
>> +DEBCACHE_ENABLED ?= "0"
>> +
>> +# Codename of the repository created by the caching class
>> +DEBDISTRONAME = "isar"
>> +
>> +# Path to the caching repository
>> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
>> +
>> +# Path to the mount point of the repository within the target
>> rootfs, during +# population
>> +DEBCACHEMNT ?= "/opt/cache/apt"
>> +
>> +# Path to the databases used by `reprepro`
>> +DEBDBDIR ?= "${LAYERDIR}/db"
>> +
>> +# Path to the configuration files templates used by `reprepro`
>> +DEBFILESDIR ?= "${LAYERDIR}/files"
>> diff --git a/meta-isar-bin/files/distributions.in
>> b/meta-isar-bin/files/distributions.in new file mode 100644
>> index 0000000..59f4429
>> --- /dev/null
>> +++ b/meta-isar-bin/files/distributions.in
>> @@ -0,0 +1,3 @@
>> +Codename: {DISTRO_NAME}
>> +Architectures: i386 armhf source
>> +Components: main
>> diff --git a/meta-isar/conf/bblayers.conf.sample
>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
>> --- a/meta-isar/conf/bblayers.conf.sample
>> +++ b/meta-isar/conf/bblayers.conf.sample
>> @@ -8,6 +8,7 @@ BBFILES ?= ""
>> BBLAYERS ?= " \
>> ##ISARROOT##/meta \
>> ##ISARROOT##/meta-isar \
>> + ##ISARROOT##/meta-isar-bin \
>> "
>> BBLAYERS_NON_REMOVABLE ?= " \
>> ##ISARROOT##/meta \
>> diff --git
>> a/meta-isar/recipes-core/images/files/debian-configscript.sh
>> b/meta-isar/recipes-core/images/files/debian-configscript.sh index
>> 4ac37d0..b05babb 100644 ---
>> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++
>> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6
>> +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2"
>> readonly ROOTFS_DEV="$3"
>> +readonly DEBCACHEMNT="$4"
>> +readonly DEBDISTRONAME="$5"
>>
>> cat >> /etc/default/locale << EOF
>> LANG=en_US.UTF-8
>> @@ -83,3 +85,9 @@ fi
>> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ];
>> then rm -f $TARGET/usr/sbin/policy-rc.d
>> fi
>> +
>> +mkdir -p /etc/apt/sources.list.d/
>> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
>> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>> +EOF
>> diff --git
>> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
>> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index
>> 2454481..f995f4d 100644 ---
>> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++
>> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@
>> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
>> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
>> +readonly DEBCACHEMNT="$4"
>> +readonly DEBDISTRONAME="$5"
>>
>> cat >> /etc/default/locale << EOF
>> LANG=en_US.UTF-8
>> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
>> cat > /boot/config.txt << EOF
>> kernel=$KERNEL_IMAGE
>> EOF
>> +
>> +mkdir -p /etc/apt/sources.list.d/
>> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
>> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>> +EOF
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>> b/meta-isar/recipes-core/images/isar-image-base.bb index
>> b679d97..85e6b27 100644 ---
>> a/meta-isar/recipes-core/images/isar-image-base.bb +++
>> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@
>> do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
>> "${WORKDIR}/multistrap.conf" || true
>> # Configure root filesystem
>> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
>> ${BAUDRATE_TTY} \
>> - ${ROOTFS_DEV}
>> + sudo chroot ${S} /configscript.sh \
>> + ${MACHINE_SERIAL} \
>> + ${BAUDRATE_TTY} \
>> + ${ROOTFS_DEV} \
>> + ${DEBCACHEMNT} \
>> + ${DEBDISTRONAME}
>> sudo rm ${S}/configscript.sh
>> }
>>
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 360a95c..b1e201d 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -1,5 +1,5 @@
>> # This software is a part of ISAR.
>> -# Copyright (C) 2015-2016 ilbers GmbH
>> +# Copyright (C) 2015-2017 ilbers GmbH
>>
>> # Add dependency from buildchroot creation
>> DEPENDS += "buildchroot"
>> @@ -55,12 +55,124 @@ do_build() {
>> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>> }
>>
>> -
>> # Install package to dedicated deploy directory
>> -do_install() {
>> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
>> +do_binary_deb_install() {
>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
>> +
>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>> + # If `bitbake` is running for the first time, the cache
>> doesn't exist
>> + # yet and needs to be configured using a `distributions`
>> file.
>> + # A template stored in the layer directory is pre-processed
>> to
>> + # generate the configuration file, which is then placed in
>> the
>> + # appropriate directory.
>> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
>> + mkdir -p "${DIR_CACHE}/conf"
>> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
>> + "${DEBFILESDIR}/distributions.in" \
>> + > "${DIR_CACHE}/conf/distributions"
>> + fi
>
> This step should be in the image and not in every package. I guess it
> has a race with multiple packages running this init routine.
Ok.
>
>> + # Add binary and source packages to the deb cache
>> + # If the cache doesn't exist yet, it will be created using
>> the
>> + # `distributions` file generated above.
>> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
>> -r p; do
>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>> "${DIR_DB}" \
>> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
>> + done
>> + else
>> + # Deb caching is disabled, simply copy all binary packages
>> to the
>> + # deploy directory
>> + mkdir -p "${DEPLOY_DIR_DEB}"
>> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
>> + fi
>> }
>>
>> -addtask install after do_build
>> -do_install[dirs] = "${DEPLOY_DIR_DEB}"
>> -do_install[stamp-extra-info] = "${MACHINE}"
>> +addtask binary_deb_install after do_build
>> +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>
> Here a task got renamed. This should be another patch ... one that i
> already tried to get in.
>
This implemented in the next patch (to keep this patch in original
format), but yes, it should be done here.
>> +
>> +# Deb caching lambda run during the parsing phase that checks
>> whether the +# current package has to be rebuilt, or taken from the
>> cache +python __anonymous () {
>> + if d.getVar("DEBCACHE_ENABLED", True) == "0":
>> + # Deb caching is disabled, do nothing
>> + return True
>> +
>> + PN = d.getVar("PN", True)
>> + PV = d.getVar("PV", True)
>> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True)
>> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
>> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True)
>> + DEBDBDIR = d.getVar("DEBDBDIR", True)
>> + DISTRO = d.getVar("DISTRO", True)
>> + path_cache = os.path.join(DEBCACHEDIR, DISTRO)
>> + path_databases = os.path.join(DEBDBDIR, DISTRO)
>> + path_distributions = os.path.join(path_cache, "conf",
>> "distributions") +
>> + # The distributions file is needed by `reprepro` to know what
>> types
>> + # of packages are supported, what the distribution name is, etc.
>> + # If it doesn't exist, we have nothing in the cache, do nothing.
>> + if not os.path.exists(path_distributions):
>> + return
>> +
>> + # Anonymous functions are run several times under different
>> contexts
>> + # during the parsing phase, which would let the code that
>> follows be run
>> + # as many times for the same package.
>> + # In order to guarantee that our subroutine only runs once per
>> package, we
>> + # use bitbake's "persist" API in order to have reliable
>> persistent storage
>> + # accross calls of the lambda (using a simple variable in the
>> class won't
>> + # work, as several contexts won't allow fetching its value).
>> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d)
>> + if PN in pd and pd[PN] == PV:
>> + return
>> +
>> + import subprocess
>> + try:
>> + # The databases used by `reprepro` are not stored within the
>> cache in
>> + # order to make versioning of only the files needed to use
>> the cache
>> + # as an official Debian repository simpler.
>> + # As such, if a developer uses a peer's cache to speed up
>> their build
>> + # time but have never run bitbake, the database will not
>> have been
>> + # created, so we regenerate them here.
>> + if not os.path.exists(path_databases) and
>> os.path.exists(path_cache):
>> + bb.note("Regenerating the cache databases...")
>> + subprocess.check_call([
>> + "reprepro",
>> + "--waitforlock", "3",
>> + "-b", path_cache,
>> + "--dbdir", path_databases,
>> + "export", DEBDISTRONAME,
>> + ])
>> +
>> + # Get a list of the versions of all the packages named after
>> the
>> + # current bitbake package, and check whether the current
>> package
>> + # version is returned.
>> + # As `reprepro` always returns zero with this particular
>> operation, we
>> + # have to use this workaround to check for a package in the
>> cache.
>> + package_version = subprocess.check_output([
>> + "reprepro",
>> + "--waitforlock", "3",
>> + "-b", path_cache,
>> + "--dbdir", path_databases,
>> + "-C", "main",
>> + "-A", DISTRO_ARCH,
>> + "--list-format", "${version}",
>> + "list", DEBDISTRONAME, PN,
>> + ])
>> + package_version = package_version.decode("utf-8")
>> + if package_version == PV:
>> + # The below list contains the names of all the tasks are
>> in charge
>> + # of building the package when the cache isn't enabled
>> or if the
>> + # package hasn't been placed in it already.
>> + # As all tasks are enabled by default, we prevent their
>> execution
>> + # by setting the `noexec` flag, which will prevent a
>> rebuild of
>> + # the package when it's cached.
>> + for task in ["fetch", "unpack", "build", "install"]:
>> + d.setVarFlag("do_{}".format(task), "noexec", "1")
>> +
>> + # Cache the results of this command so that subsequent
>> executions
>> + # of this anonymous functions don't run the same code
>> again.
>> + pd[PN] = PV
>> + except subprocess.CalledProcessError as e:
>> + bb.fatal("Unable to check for a candidate for package {0}
>> (errorcode: {1})".format(PN, e.returncode)) +}
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index a7f0d74..f42aa48 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
>>
>> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>
>> -# Install Debian packages, that were built from sources
>> +# Install Debian packages from the cache
>> do_populate() {
>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>> +
>> if [ -n "${IMAGE_INSTALL}" ]; then
>> - sudo mkdir -p ${S}/deb
>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>> + sudo mkdir -p "${S}/${DEBCACHEMNT}"
>> + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
>> +
>> + sudo chroot "${S}" apt-get update -y
>
> This will fail if you are behind a proxy. It makes a known issue worse.
>
>> + for package in ${IMAGE_INSTALL}; do
>> + sudo chroot "${S}" apt-get install -t
>> "${DEBDISTRONAME}" -y \
>> + --allow-unauthenticated "${package}"
>> + done
>> +
>> + sudo umount "${S}/${DEBCACHEMNT}"
>
> A whole lot of sudo. I do not mind but am still wondering ...
Please questions and proposals instead of negative tone.
>
> Can we replace the bind-mount with a symlink? If anything between the
> mount and umount fails and you need to run again ... will the mount
> fail if the old one is still there?
If we switch to multistrap-based populating, I think this will go.
>
>> + else
>> + sudo mkdir -p ${S}/deb
>>
>> - for p in ${IMAGE_INSTALL}; do
>> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
>> - done
>> + for p in ${IMAGE_INSTALL}; do
>> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec
>> \
>> + sudo cp '{}' "${S}/deb/" \;
>> + done
>>
>> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb
>> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb
>>
>> - sudo rm -rf ${S}/deb
>> + sudo rm -rf ${S}/deb
>> + fi
>> fi
>> }
>>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache
2017-08-28 8:00 ` Claudius Heine
@ 2017-08-29 7:18 ` Alexander Smirnov
2017-08-30 8:54 ` Claudius Heine
0 siblings, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-29 7:18 UTC (permalink / raw)
To: Claudius Heine, isar-users
On 08/28/2017 11:00 AM, Claudius Heine wrote:
> Hi,
>
> On 08/27/2017 05:13 PM, Alexander Smirnov wrote:
>> Split install function into two parts:
>> - install_to_cache: if caching is enabled
>> - install_to_deploy: if caching is disabled
>>
>> This patch brings flexibility to the implementation and makes it possible
>> to move all the caching implementation code to dedicated class. The magic
>> behavior depending on the value of DEBCACHE_ENABLED is now transparent in
>> the bitbake pipeline (can be inspected via "bitbake -g").
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta/classes/dpkg.bbclass | 66
>> ++++++++++++++++++++++++++--------------------
>> meta/classes/image.bbclass | 2 +-
>> 2 files changed, 38 insertions(+), 30 deletions(-)
>>
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index b1e201d..118ba2f 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -56,46 +56,53 @@ do_build() {
>> }
>> # Install package to dedicated deploy directory
>> -do_binary_deb_install() {
>> +do_install_to_deploy() {
>> + # Deb caching is disabled, simply copy all binary packages to the
>> deploy
>> + # directory
>> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
>> +}
>> +
>> +addtask install_to_deploy after do_build
>> +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}"
>> +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> +do_install_to_deploy[noexec] = "1"
>> +
>> +# Install package to dedicated apt repo
>> +do_install_to_cache() {
>> readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>> readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
>> - if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>> - # If `bitbake` is running for the first time, the cache
>> doesn't exist
>> - # yet and needs to be configured using a `distributions` file.
>> - # A template stored in the layer directory is pre-processed to
>> - # generate the configuration file, which is then placed in the
>> - # appropriate directory.
>> - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
>> - mkdir -p "${DIR_CACHE}/conf"
>> - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
>> - "${DEBFILESDIR}/distributions.in" \
>> - > "${DIR_CACHE}/conf/distributions"
>> - fi
>> -
>> - # Add binary and source packages to the deb cache
>> - # If the cache doesn't exist yet, it will be created using the
>> - # `distributions` file generated above.
>> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
>> -r p; do
>> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>> "${DIR_DB}" \
>> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
>> - done
>> - else
>> - # Deb caching is disabled, simply copy all binary packages to
>> the
>> - # deploy directory
>> - mkdir -p "${DEPLOY_DIR_DEB}"
>> - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
>> + # If `bitbake` is running for the first time, the cache doesn't
>> exist
>> + # yet and needs to be configured using a `distributions` file.
>> + # A template stored in the layer directory is pre-processed to
>> + # generate the configuration file, which is then placed in the
>> + # appropriate directory.
>> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
>> + mkdir -p "${DIR_CACHE}/conf"
>> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
>> + "${DEBFILESDIR}/distributions.in" \
>> + > "${DIR_CACHE}/conf/distributions"
>> fi
>> +
>> + # Add binary and source packages to the deb cache
>> + # If the cache doesn't exist yet, it will be created using the
>> + # `distributions` file generated above.
>> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r
>> p; do
>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir "${DIR_DB}" \
>> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
>> + done
>> }
>> -addtask binary_deb_install after do_build
>> -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> +addtask install_to_cache after do_build
>
> What was the reason why we put stuff after the 'build' task? In OE the
> 'build' task is the default task triggering everything necessary for a
> recipe. In Isar that changed. Why?
Actually AFAIK do_build is not OE tasks, it's default bitbake task.
Install is after do_build, because in Isar do_build is used to build the
packages, so it's not empty.
>
>> +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>> +do_install_to_cache[noexec] = "1"
>> # Deb caching lambda run during the parsing phase that checks
>> whether the
>> # current package has to be rebuilt, or taken from the cache
>> python __anonymous () {
>> if d.getVar("DEBCACHE_ENABLED", True) == "0":
>> - # Deb caching is disabled, do nothing
>> + # Deb caching is disabled
>> + d.delVarFlag('do_install_to_deploy', 'noexec')
>> return True
>> PN = d.getVar("PN", True)
>> @@ -108,6 +115,7 @@ python __anonymous () {
>> path_cache = os.path.join(DEBCACHEDIR, DISTRO)
>> path_databases = os.path.join(DEBDBDIR, DISTRO)
>> path_distributions = os.path.join(path_cache, "conf",
>> "distributions")
>> + d.delVarFlag('do_install_to_cache', 'noexec')
>> # The distributions file is needed by `reprepro` to know what types
>> # of packages are supported, what the distribution name is, etc.
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index c2ff453..6b1b5eb 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -46,4 +46,4 @@ do_populate() {
>> }
>> addtask populate before do_build
>> -do_populate[deptask] = "do_install"
>> +do_populate[deptask] = "do_install_to_cache do_install_to_deploy"
>
> I would rather still have a, maybe empty, 'do_install' step that just
> depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks. I
> don't see why ever small change in one class should change the complete
> pipeline for every recipe.
It's a bit difficult question. In general meaning when building
something from sources, the install means a storing of build artifacts
to some specific location. So in this case, we just copy binary deb
package to repo, and there is nothing common with installation of artifacts.
Also Isar has in roadmap cross-compilation, so the pipeline will
probably look like:
fetch - unpack - compile - install - install_to_cache
so in this case the install task probably should go before install_to_cache.
So probably it makes sense to avoid _install_ in this task, for example
use do_populate_deb_cache, what do you think?
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache
2017-08-28 15:32 ` Henning Schild
@ 2017-08-29 7:20 ` Alexander Smirnov
2017-08-29 7:57 ` Henning Schild
0 siblings, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-29 7:20 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
> This is another fixup of 1/6, fold it in.
>
It's a good question if we want to keep development history. Because
these patches reflect the real history of binary caching development.
>
> Am Sun, 27 Aug 2017 18:13:38 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> From: Frank Lenormand <lenormf@gmail.com>
>>
>> Before installing to cache, existing packages with the same name and
>> version have to be removed.
>>
>> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta/classes/dpkg.bbclass | 46
>> +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43
>> insertions(+), 3 deletions(-)
>>
>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>> index 118ba2f..72ae79f 100644
>> --- a/meta/classes/dpkg.bbclass
>> +++ b/meta/classes/dpkg.bbclass
>> @@ -84,12 +84,52 @@ do_install_to_cache() {
>> > "${DIR_CACHE}/conf/distributions"
>> fi
>>
>> + print_field_value() {
>> + awk "\$1 == \"${1}:\" { print \$2; }"
>> + }
>> +
>> + call_reprepro() {
>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>> "${DIR_DB}" \
>> + -C main "$@"
>> + }
>> +
>> # Add binary and source packages to the deb cache
>> # If the cache doesn't exist yet, it will be created using the
>> # `distributions` file generated above.
>> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r
>> p; do
>> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>> "${DIR_DB}" \
>> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
>> + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do
>> + name_package=$(dpkg -f "${p}" | print_field_value "Package")
>> + version_package=$(dpkg -f "${p}" | print_field_value
>> "Version") +
>> + # Remove all packages with the same version that were added
>> to the
>> + # repository in previous builds
>> + call_reprepro \
>> + -A "${DISTRO_ARCH}" \
>> + removefilter "${DEBDISTRONAME}" \
>> + "Package (== ${name_package}), \
>> + Version (== ${version_package})"
>> + call_reprepro \
>> + -A "${DISTRO_ARCH}" \
>> + "include${p##*.}" \
>> + "${DEBDISTRONAME}" \
>> + "${p}"
>> + done
>> +
>> + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do
>> + name_package=$(cat "${p}" | print_field_value "Source")
>> + version_package=$(cat "${p}" | print_field_value "Version")
>> +
>> + # Remove all source packages with the same version that were
>> added to
>> + # the repository in previous builds
>> + call_reprepro \
>> + -A "source" \
>> + removefilter "${DEBDISTRONAME}" \
>> + "Package (== ${name_package}), \
>> + Version (== ${version_package})"
>> + call_reprepro \
>> + -A "source" \
>> + "include${p##*.}" \
>> + "${DEBDISTRONAME}" \
>> + "${p}"
>> done
>> }
>>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache
2017-08-28 15:36 ` Henning Schild
@ 2017-08-29 7:29 ` Alexander Smirnov
2017-08-29 8:06 ` Henning Schild
2017-08-29 11:29 ` Jan Kiszka
1 sibling, 1 reply; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-29 7:29 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
> This should be done along the way in every single patch, not afterwards.
I don't think that it really should. The requirement to documentation is
to describe current code state in repo. So if something has been changed
in design - HEAD of branch should contain relevant documentation, what
could be achieved by dedicate patch. Moreover this single patch
explicitly describes the influence of whole series to current design.
>
> Henning
>
> Am Sun, 27 Aug 2017 18:13:39 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> Describe Isar binary cache feature.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> doc/technical_overview.md | 59
>> +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55
>> insertions(+), 4 deletions(-)
>>
>> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
>> index 20e08b4..b3265dc 100644
>> --- a/doc/technical_overview.md
>> +++ b/doc/technical_overview.md
>> @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described as
>> following:
>> - According to the list of custom packages in bitbake recipes, the
>> initial filesystem will be populated by successfully built packages.
>>
>> +## 2.5 Binary Cache
>> +
>> +As mentioned above, apart from fetching binary packages from
>> upstream apt +repositories, Isar is able to build custom packages
>> from sources. To simplify +further custom packages management, Isar
>> implements binary caching. +
>> +Binary cache is a typical apt repository created by Isar to store
>> newly built +packages. This repository has mainly two goals:
>> +
>> + - Simplify image filesystem populating and distributing binary
>> packages to
>> + other systems.
>> +
>> + - Reuse these binaries in next builds to avoid re-building from the
>> + scratch everytime.
>> +
>> +Current Isar tree provides default cache repository: `meta-isar-bin`.
>> +Isar binary cache is described by the following files:
>> +
>> + - `meta-isar-bin/conf/layer.conf`
>> +
>> + - `meta-isar-bin/files/distributions.in`
>> +
>> +After system build is complete, this repo contains binary packages
>> for all the +distros and architectures, requested in multiconfig. So
>> this folder can be +published and used as apt repo for already
>> installed systems. +
>> +An important note, that Isar creates local repositories using
>> dedicated Debian +tool reprepro. This tool takes care about:
>> +
>> + - Maintenance of multiple architectures and distros.
>> +
>> + - Internal packages database management.
>> +
>> + - Repo signatures management.
>> +
>> + - Requests about available packages in repo (name, version etc...).
>> +
>> # 3 Isar Internal Processes
>>
>> ## 3.1 General Overview
>> @@ -174,10 +211,17 @@ contain debian folder. The build process is
>> implemented in
>> 3. Run dpkg-buildpackage
>>
>> -4. Task `do_install`: install successfully built packages
>> +4. Further package processing depends on binary cache option:
>> +
>> + 1. Task `do_install_to_deploy`: install successfully built
>> packages if
>> + binary cache is disabled:
>> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy directory
>> `${DEPLOY_DIR_DEB}`
>>
>> + 2. Task `do_install_to_cache`: install successfully built
>> packages to
>> + binary cache repository:
>> + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}`
>> +
>> ## 3.5 Populate Target Filesystem
>>
>> Each target image can be extended by custom packages listed in
>> IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate`
>> performs the following:
>> 1. Parse IMAGE_INSTALL variable.
>>
>> -2. Find respective packages in `${DEPLOY_DIR_DEB}`.
>> +2. Further steps depends in binary cache option.
>> +If binary cache is disabled:
>> +
>> + 1. Find respective packages in `${DEPLOY_DIR_DEB}`.
>> +
>> + 2. Copy packages to deb folder in dedicated target filesystem.
>> +
>> + 3. Execute dpkg command in chroot for all the copied packages.
>>
>> -3. Copy them to deb folder in dedicated target filesystem.
>> +If binary cache is enabled:
>>
>> -4. Execute dpkg command in chroot for all the copied packages.
>> + 1. Call apt-get install in chroot in target image filesystem.
>>
>> ## 3.6 Generate Bootable Image
>>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-29 6:40 ` Alexander Smirnov
@ 2017-08-29 7:51 ` Henning Schild
2017-08-29 8:20 ` Alexander Smirnov
0 siblings, 1 reply; 30+ messages in thread
From: Henning Schild @ 2017-08-29 7:51 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Tue, 29 Aug 2017 09:40:46 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> On 08/28/2017 06:18 PM, Henning Schild wrote:
> > This patch is very big, doing a lot of things at a time.
>
> Ok, I'll think how it can be split.
>
> > The commit message on the other hand is not very verbose.
>
> Half of the patch is the comments that describe in details what
> happens in the code, so please questions and proposals.
I will comment on that once the patch got split. In general readable
good does not require comments. Comments add redundancy and if comments
are required it is usually a sign that the code is too complicated.
> >
> > creating the repo is one step
> > - i think using reprepro is a good idea
> > moving do_populate to that repo is another one
> > - i think this task should go away and the repo should be given to
> > multistrap
>
> Ok
>
> > and there are probably more ways to split that into reviewable
> > pieces
>
> Please questions and proposals.
Yes, for v2.
> >
> > So this review is not too verbose, we need smaller pieces.
> >
> > More inline
> >
> > Am Sun, 27 Aug 2017 18:13:34 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >
> >> From: Frank Lenormand <lenormf@gmail.com>
> >>
> >> Adds the possibility to create an apt repository for newly built
> >> packages and then use that repo for populating the target
> >> filesystem.
> >>
> >> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >> meta-isar-bin/conf/layer.conf | 18 +++
> >> meta-isar-bin/files/distributions.in | 3 +
> >> meta-isar/conf/bblayers.conf.sample | 1 +
> >> .../images/files/debian-configscript.sh | 8 ++
> >> .../images/files/raspbian-configscript.sh | 8 ++
> >> meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
> >> meta/classes/dpkg.bbclass | 126
> >> +++++++++++++++++++--
> >> meta/classes/image.bbclass | 30 +++-- 8
> >> files changed, 186 insertions(+), 16 deletions(-) create mode
> >> 100644 meta-isar-bin/conf/layer.conf create mode 100644
> >> meta-isar-bin/files/distributions.in
> >>
> >> diff --git a/meta-isar-bin/conf/layer.conf
> >> b/meta-isar-bin/conf/layer.conf new file mode 100644
> >> index 0000000..24534e1
> >> --- /dev/null
> >> +++ b/meta-isar-bin/conf/layer.conf
> >> @@ -0,0 +1,18 @@
> >> +# Enable package caching with '1'
> >> +DEBCACHE_ENABLED ?= "0"
> >> +
> >> +# Codename of the repository created by the caching class
> >> +DEBDISTRONAME = "isar"
> >> +
> >> +# Path to the caching repository
> >> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> >> +
> >> +# Path to the mount point of the repository within the target
> >> rootfs, during +# population
> >> +DEBCACHEMNT ?= "/opt/cache/apt"
> >> +
> >> +# Path to the databases used by `reprepro`
> >> +DEBDBDIR ?= "${LAYERDIR}/db"
> >> +
> >> +# Path to the configuration files templates used by `reprepro`
> >> +DEBFILESDIR ?= "${LAYERDIR}/files"
> >> diff --git a/meta-isar-bin/files/distributions.in
> >> b/meta-isar-bin/files/distributions.in new file mode 100644
> >> index 0000000..59f4429
> >> --- /dev/null
> >> +++ b/meta-isar-bin/files/distributions.in
> >> @@ -0,0 +1,3 @@
> >> +Codename: {DISTRO_NAME}
> >> +Architectures: i386 armhf source
> >> +Components: main
> >> diff --git a/meta-isar/conf/bblayers.conf.sample
> >> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> >> --- a/meta-isar/conf/bblayers.conf.sample
> >> +++ b/meta-isar/conf/bblayers.conf.sample
> >> @@ -8,6 +8,7 @@ BBFILES ?= ""
> >> BBLAYERS ?= " \
> >> ##ISARROOT##/meta \
> >> ##ISARROOT##/meta-isar \
> >> + ##ISARROOT##/meta-isar-bin \
> >> "
> >> BBLAYERS_NON_REMOVABLE ?= " \
> >> ##ISARROOT##/meta \
> >> diff --git
> >> a/meta-isar/recipes-core/images/files/debian-configscript.sh
> >> b/meta-isar/recipes-core/images/files/debian-configscript.sh index
> >> 4ac37d0..b05babb 100644 ---
> >> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++
> >> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@
> >> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
> >> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
> >> +readonly DEBCACHEMNT="$4"
> >> +readonly DEBDISTRONAME="$5"
> >>
> >> cat >> /etc/default/locale << EOF
> >> LANG=en_US.UTF-8
> >> @@ -83,3 +85,9 @@ fi
> >> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d"
> >> ]; then rm -f $TARGET/usr/sbin/policy-rc.d
> >> fi
> >> +
> >> +mkdir -p /etc/apt/sources.list.d/
> >> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
> >> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> >> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> >> +EOF
> >> diff --git
> >> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
> >> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
> >> index 2454481..f995f4d 100644 ---
> >> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++
> >> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@
> >> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
> >> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
> >> +readonly DEBCACHEMNT="$4"
> >> +readonly DEBDISTRONAME="$5"
> >>
> >> cat >> /etc/default/locale << EOF
> >> LANG=en_US.UTF-8
> >> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
> >> cat > /boot/config.txt << EOF
> >> kernel=$KERNEL_IMAGE
> >> EOF
> >> +
> >> +mkdir -p /etc/apt/sources.list.d/
> >> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
> >> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> >> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> >> +EOF
> >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> >> b/meta-isar/recipes-core/images/isar-image-base.bb index
> >> b679d97..85e6b27 100644 ---
> >> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> >> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12
> >> @@ do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> >> "${WORKDIR}/multistrap.conf" || true
> >> # Configure root filesystem
> >> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> >> ${BAUDRATE_TTY} \
> >> - ${ROOTFS_DEV}
> >> + sudo chroot ${S} /configscript.sh \
> >> + ${MACHINE_SERIAL} \
> >> + ${BAUDRATE_TTY} \
> >> + ${ROOTFS_DEV} \
> >> + ${DEBCACHEMNT} \
> >> + ${DEBDISTRONAME}
> >> sudo rm ${S}/configscript.sh
> >> }
> >>
> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> >> index 360a95c..b1e201d 100644
> >> --- a/meta/classes/dpkg.bbclass
> >> +++ b/meta/classes/dpkg.bbclass
> >> @@ -1,5 +1,5 @@
> >> # This software is a part of ISAR.
> >> -# Copyright (C) 2015-2016 ilbers GmbH
> >> +# Copyright (C) 2015-2017 ilbers GmbH
> >>
> >> # Add dependency from buildchroot creation
> >> DEPENDS += "buildchroot"
> >> @@ -55,12 +55,124 @@ do_build() {
> >> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
> >> }
> >>
> >> -
> >> # Install package to dedicated deploy directory
> >> -do_install() {
> >> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
> >> +do_binary_deb_install() {
> >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> >> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
> >> +
> >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> >> + # If `bitbake` is running for the first time, the cache
> >> doesn't exist
> >> + # yet and needs to be configured using a `distributions`
> >> file.
> >> + # A template stored in the layer directory is
> >> pre-processed to
> >> + # generate the configuration file, which is then placed in
> >> the
> >> + # appropriate directory.
> >> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> >> + mkdir -p "${DIR_CACHE}/conf"
> >> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> >> + "${DEBFILESDIR}/distributions.in" \
> >> + > "${DIR_CACHE}/conf/distributions"
> >> + fi
> >
> > This step should be in the image and not in every package. I guess
> > it has a race with multiple packages running this init routine.
>
> Ok.
>
> >
> >> + # Add binary and source packages to the deb cache
> >> + # If the cache doesn't exist yet, it will be created using
> >> the
> >> + # `distributions` file generated above.
> >> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while
> >> read -r p; do
> >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> >> "${DIR_DB}" \
> >> + -C main "include${p##*.}" "${DEBDISTRONAME}"
> >> "${p}"
> >> + done
> >> + else
> >> + # Deb caching is disabled, simply copy all binary packages
> >> to the
> >> + # deploy directory
> >> + mkdir -p "${DEPLOY_DIR_DEB}"
> >> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> >> + fi
> >> }
> >>
> >> -addtask install after do_build
> >> -do_install[dirs] = "${DEPLOY_DIR_DEB}"
> >> -do_install[stamp-extra-info] = "${MACHINE}"
> >> +addtask binary_deb_install after do_build
> >> +do_binary_deb_install[stamp-extra-info] =
> >> "${DISTRO}-${DISTRO_ARCH}"
> >
> > Here a task got renamed. This should be another patch ... one that i
> > already tried to get in.
> >
>
> This implemented in the next patch (to keep this patch in original
> format), but yes, it should be done here.
>
> >> +
> >> +# Deb caching lambda run during the parsing phase that checks
> >> whether the +# current package has to be rebuilt, or taken from the
> >> cache +python __anonymous () {
> >> + if d.getVar("DEBCACHE_ENABLED", True) == "0":
> >> + # Deb caching is disabled, do nothing
> >> + return True
> >> +
> >> + PN = d.getVar("PN", True)
> >> + PV = d.getVar("PV", True)
> >> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True)
> >> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
> >> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True)
> >> + DEBDBDIR = d.getVar("DEBDBDIR", True)
> >> + DISTRO = d.getVar("DISTRO", True)
> >> + path_cache = os.path.join(DEBCACHEDIR, DISTRO)
> >> + path_databases = os.path.join(DEBDBDIR, DISTRO)
> >> + path_distributions = os.path.join(path_cache, "conf",
> >> "distributions") +
> >> + # The distributions file is needed by `reprepro` to know what
> >> types
> >> + # of packages are supported, what the distribution name is,
> >> etc.
> >> + # If it doesn't exist, we have nothing in the cache, do
> >> nothing.
> >> + if not os.path.exists(path_distributions):
> >> + return
> >> +
> >> + # Anonymous functions are run several times under different
> >> contexts
> >> + # during the parsing phase, which would let the code that
> >> follows be run
> >> + # as many times for the same package.
> >> + # In order to guarantee that our subroutine only runs once per
> >> package, we
> >> + # use bitbake's "persist" API in order to have reliable
> >> persistent storage
> >> + # accross calls of the lambda (using a simple variable in the
> >> class won't
> >> + # work, as several contexts won't allow fetching its value).
> >> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d)
> >> + if PN in pd and pd[PN] == PV:
> >> + return
> >> +
> >> + import subprocess
> >> + try:
> >> + # The databases used by `reprepro` are not stored within
> >> the cache in
> >> + # order to make versioning of only the files needed to use
> >> the cache
> >> + # as an official Debian repository simpler.
> >> + # As such, if a developer uses a peer's cache to speed up
> >> their build
> >> + # time but have never run bitbake, the database will not
> >> have been
> >> + # created, so we regenerate them here.
> >> + if not os.path.exists(path_databases) and
> >> os.path.exists(path_cache):
> >> + bb.note("Regenerating the cache databases...")
> >> + subprocess.check_call([
> >> + "reprepro",
> >> + "--waitforlock", "3",
> >> + "-b", path_cache,
> >> + "--dbdir", path_databases,
> >> + "export", DEBDISTRONAME,
> >> + ])
> >> +
> >> + # Get a list of the versions of all the packages named
> >> after the
> >> + # current bitbake package, and check whether the current
> >> package
> >> + # version is returned.
> >> + # As `reprepro` always returns zero with this particular
> >> operation, we
> >> + # have to use this workaround to check for a package in
> >> the cache.
> >> + package_version = subprocess.check_output([
> >> + "reprepro",
> >> + "--waitforlock", "3",
> >> + "-b", path_cache,
> >> + "--dbdir", path_databases,
> >> + "-C", "main",
> >> + "-A", DISTRO_ARCH,
> >> + "--list-format", "${version}",
> >> + "list", DEBDISTRONAME, PN,
> >> + ])
> >> + package_version = package_version.decode("utf-8")
> >> + if package_version == PV:
> >> + # The below list contains the names of all the tasks
> >> are in charge
> >> + # of building the package when the cache isn't enabled
> >> or if the
> >> + # package hasn't been placed in it already.
> >> + # As all tasks are enabled by default, we prevent
> >> their execution
> >> + # by setting the `noexec` flag, which will prevent a
> >> rebuild of
> >> + # the package when it's cached.
> >> + for task in ["fetch", "unpack", "build", "install"]:
> >> + d.setVarFlag("do_{}".format(task), "noexec", "1")
> >> +
> >> + # Cache the results of this command so that subsequent
> >> executions
> >> + # of this anonymous functions don't run the same code
> >> again.
> >> + pd[PN] = PV
> >> + except subprocess.CalledProcessError as e:
> >> + bb.fatal("Unable to check for a candidate for package {0}
> >> (errorcode: {1})".format(PN, e.returncode)) +}
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index a7f0d74..f42aa48 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
> >>
> >> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
> >>
> >> -# Install Debian packages, that were built from sources
> >> +# Install Debian packages from the cache
> >> do_populate() {
> >> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> >> +
> >> if [ -n "${IMAGE_INSTALL}" ]; then
> >> - sudo mkdir -p ${S}/deb
> >> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> >> + sudo mkdir -p "${S}/${DEBCACHEMNT}"
> >> + sudo mount -o bind "${DIR_CACHE}"
> >> "${S}/${DEBCACHEMNT}" +
> >> + sudo chroot "${S}" apt-get update -y
> >
> > This will fail if you are behind a proxy. It makes a known issue
> > worse.
> >> + for package in ${IMAGE_INSTALL}; do
> >> + sudo chroot "${S}" apt-get install -t
> >> "${DEBDISTRONAME}" -y \
> >> + --allow-unauthenticated "${package}"
> >> + done
> >> +
> >> + sudo umount "${S}/${DEBCACHEMNT}"
> >
> > A whole lot of sudo. I do not mind but am still wondering ...
>
> Please questions and proposals instead of negative tone.
That was maybe slightly negative. Why? Because that patch was sent at a
time where the policy was still "do not add more sudos". And the patch
violates more of the groundrules we have agreed on.
You posting the patch signals that our groundrules seem to apply
to !ilbers-contributions only.
> > Can we replace the bind-mount with a symlink? If anything between
> > the mount and umount fails and you need to run again ... will the
> > mount fail if the old one is still there?
>
> If we switch to multistrap-based populating, I think this will go.
I know, but we have not decided that yet.
Henning
> >
> >> + else
> >> + sudo mkdir -p ${S}/deb
> >>
> >> - for p in ${IMAGE_INSTALL}; do
> >> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
> >> - done
> >> + for p in ${IMAGE_INSTALL}; do
> >> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb'
> >> -exec \
> >> + sudo cp '{}' "${S}/deb/" \;
> >> + done
> >>
> >> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb
> >> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb
> >>
> >> - sudo rm -rf ${S}/deb
> >> + sudo rm -rf ${S}/deb
> >> + fi
> >> fi
> >> }
> >>
> >
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache
2017-08-29 7:20 ` Alexander Smirnov
@ 2017-08-29 7:57 ` Henning Schild
2017-08-29 11:26 ` Jan Kiszka
0 siblings, 1 reply; 30+ messages in thread
From: Henning Schild @ 2017-08-29 7:57 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Tue, 29 Aug 2017 10:20:41 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> > This is another fixup of 1/6, fold it in.
> >
>
> It's a good question if we want to keep development history. Because
> these patches reflect the real history of binary caching development.
To me the answer is clear. If we find issues in a patch during review,
the patch should be fixed before getting merged.
If we do not follow that policy you might as well merge all patches
without review.
Henning
> >
> > Am Sun, 27 Aug 2017 18:13:38 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >
> >> From: Frank Lenormand <lenormf@gmail.com>
> >>
> >> Before installing to cache, existing packages with the same name
> >> and version have to be removed.
> >>
> >> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >> meta/classes/dpkg.bbclass | 46
> >> +++++++++++++++++++++++++++++++++++++++++++--- 1 file changed, 43
> >> insertions(+), 3 deletions(-)
> >>
> >> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> >> index 118ba2f..72ae79f 100644
> >> --- a/meta/classes/dpkg.bbclass
> >> +++ b/meta/classes/dpkg.bbclass
> >> @@ -84,12 +84,52 @@ do_install_to_cache() {
> >> > "${DIR_CACHE}/conf/distributions"
> >> fi
> >>
> >> + print_field_value() {
> >> + awk "\$1 == \"${1}:\" { print \$2; }"
> >> + }
> >> +
> >> + call_reprepro() {
> >> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> >> "${DIR_DB}" \
> >> + -C main "$@"
> >> + }
> >> +
> >> # Add binary and source packages to the deb cache
> >> # If the cache doesn't exist yet, it will be created using
> >> the # `distributions` file generated above.
> >> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
> >> -r p; do
> >> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> >> "${DIR_DB}" \
> >> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> >> + ls -1 "${BUILDROOT}"/*.deb | while read -r p; do
> >> + name_package=$(dpkg -f "${p}" | print_field_value
> >> "Package")
> >> + version_package=$(dpkg -f "${p}" | print_field_value
> >> "Version") +
> >> + # Remove all packages with the same version that were
> >> added to the
> >> + # repository in previous builds
> >> + call_reprepro \
> >> + -A "${DISTRO_ARCH}" \
> >> + removefilter "${DEBDISTRONAME}" \
> >> + "Package (== ${name_package}), \
> >> + Version (== ${version_package})"
> >> + call_reprepro \
> >> + -A "${DISTRO_ARCH}" \
> >> + "include${p##*.}" \
> >> + "${DEBDISTRONAME}" \
> >> + "${p}"
> >> + done
> >> +
> >> + ls -1 "${BUILDROOT}"/*.dsc | while read -r p; do
> >> + name_package=$(cat "${p}" | print_field_value "Source")
> >> + version_package=$(cat "${p}" | print_field_value
> >> "Version") +
> >> + # Remove all source packages with the same version that
> >> were added to
> >> + # the repository in previous builds
> >> + call_reprepro \
> >> + -A "source" \
> >> + removefilter "${DEBDISTRONAME}" \
> >> + "Package (== ${name_package}), \
> >> + Version (== ${version_package})"
> >> + call_reprepro \
> >> + -A "source" \
> >> + "include${p##*.}" \
> >> + "${DEBDISTRONAME}" \
> >> + "${p}"
> >> done
> >> }
> >>
> >
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache
2017-08-29 7:29 ` Alexander Smirnov
@ 2017-08-29 8:06 ` Henning Schild
0 siblings, 0 replies; 30+ messages in thread
From: Henning Schild @ 2017-08-29 8:06 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Tue, 29 Aug 2017 10:29:15 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> > This should be done along the way in every single patch, not
> > afterwards.
>
> I don't think that it really should. The requirement to documentation
> is to describe current code state in repo. So if something has been
> changed in design - HEAD of branch should contain relevant
> documentation, what could be achieved by dedicate patch. Moreover
> this single patch explicitly describes the influence of whole series
> to current design.
Every commit should leave the whole repo in a consistent state. If you
change something and change the doc in a later commit you have
inconsistent intermediate states. I already explained that to you in
detail when we talked about the order in which the document should
appear in the repo.
You can argue that humans can deal with such inconsistencies, true.
Say there was a change like that, in a language that a machine reads:
+ mount /foo
+ umount /foo
Would you want those two lines in one or in two patches? If two are OK,
should they be merge directly after another or can we put 10 more
between them?
I think the answer is clear, since you want CI to succeed on every
commit. So you want every commit to be consistent in terms of
programming language. And i am saying we should do the same for
natural/human language.
Henning
> >
> > Henning
> >
> > Am Sun, 27 Aug 2017 18:13:39 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >
> >> Describe Isar binary cache feature.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >> doc/technical_overview.md | 59
> >> +++++++++++++++++++++++++++++++++++++++++++---- 1 file changed, 55
> >> insertions(+), 4 deletions(-)
> >>
> >> diff --git a/doc/technical_overview.md b/doc/technical_overview.md
> >> index 20e08b4..b3265dc 100644
> >> --- a/doc/technical_overview.md
> >> +++ b/doc/technical_overview.md
> >> @@ -100,6 +100,43 @@ Target filesystem lifecycle can be described
> >> as following:
> >> - According to the list of custom packages in bitbake recipes,
> >> the initial filesystem will be populated by successfully built
> >> packages.
> >> +## 2.5 Binary Cache
> >> +
> >> +As mentioned above, apart from fetching binary packages from
> >> upstream apt +repositories, Isar is able to build custom packages
> >> from sources. To simplify +further custom packages management, Isar
> >> implements binary caching. +
> >> +Binary cache is a typical apt repository created by Isar to store
> >> newly built +packages. This repository has mainly two goals:
> >> +
> >> + - Simplify image filesystem populating and distributing binary
> >> packages to
> >> + other systems.
> >> +
> >> + - Reuse these binaries in next builds to avoid re-building from
> >> the
> >> + scratch everytime.
> >> +
> >> +Current Isar tree provides default cache repository:
> >> `meta-isar-bin`. +Isar binary cache is described by the following
> >> files: +
> >> + - `meta-isar-bin/conf/layer.conf`
> >> +
> >> + - `meta-isar-bin/files/distributions.in`
> >> +
> >> +After system build is complete, this repo contains binary packages
> >> for all the +distros and architectures, requested in multiconfig.
> >> So this folder can be +published and used as apt repo for already
> >> installed systems. +
> >> +An important note, that Isar creates local repositories using
> >> dedicated Debian +tool reprepro. This tool takes care about:
> >> +
> >> + - Maintenance of multiple architectures and distros.
> >> +
> >> + - Internal packages database management.
> >> +
> >> + - Repo signatures management.
> >> +
> >> + - Requests about available packages in repo (name, version
> >> etc...). +
> >> # 3 Isar Internal Processes
> >>
> >> ## 3.1 General Overview
> >> @@ -174,10 +211,17 @@ contain debian folder. The build process is
> >> implemented in
> >> 3. Run dpkg-buildpackage
> >>
> >> -4. Task `do_install`: install successfully built packages
> >> +4. Further package processing depends on binary cache option:
> >> +
> >> + 1. Task `do_install_to_deploy`: install successfully built
> >> packages if
> >> + binary cache is disabled:
> >> `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to deploy
> >> directory `${DEPLOY_DIR_DEB}`
> >>
> >> + 2. Task `do_install_to_cache`: install successfully built
> >> packages to
> >> + binary cache repository:
> >> + `${BUILDCHROOT_DIR}/home/build/${PN}/*.deb` to `${DIR_CACHE}`
> >> +
> >> ## 3.5 Populate Target Filesystem
> >>
> >> Each target image can be extended by custom packages listed in
> >> IMAGE_INSTALL @@ -185,11 +229,18 @@ variable. Task `do_populate`
> >> performs the following:
> >> 1. Parse IMAGE_INSTALL variable.
> >>
> >> -2. Find respective packages in `${DEPLOY_DIR_DEB}`.
> >> +2. Further steps depends in binary cache option.
> >> +If binary cache is disabled:
> >> +
> >> + 1. Find respective packages in `${DEPLOY_DIR_DEB}`.
> >> +
> >> + 2. Copy packages to deb folder in dedicated target filesystem.
> >> +
> >> + 3. Execute dpkg command in chroot for all the copied packages.
> >>
> >> -3. Copy them to deb folder in dedicated target filesystem.
> >> +If binary cache is enabled:
> >>
> >> -4. Execute dpkg command in chroot for all the copied packages.
> >> + 1. Call apt-get install in chroot in target image filesystem.
> >>
> >> ## 3.6 Generate Bootable Image
> >>
> >
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-29 7:51 ` Henning Schild
@ 2017-08-29 8:20 ` Alexander Smirnov
0 siblings, 0 replies; 30+ messages in thread
From: Alexander Smirnov @ 2017-08-29 8:20 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 08/29/2017 10:51 AM, Henning Schild wrote:
> Am Tue, 29 Aug 2017 09:40:46 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> On 08/28/2017 06:18 PM, Henning Schild wrote:
>>> This patch is very big, doing a lot of things at a time.
>>
>> Ok, I'll think how it can be split.
>>
>>> The commit message on the other hand is not very verbose.
>>
>> Half of the patch is the comments that describe in details what
>> happens in the code, so please questions and proposals.
>
> I will comment on that once the patch got split. In general readable
> good does not require comments. Comments add redundancy and if comments
> are required it is usually a sign that the code is too complicated.
I think it's a matter of taste, but Ok, I got your position.
>
>>>
>>> creating the repo is one step
>>> - i think using reprepro is a good idea
>>> moving do_populate to that repo is another one
>>> - i think this task should go away and the repo should be given to
>>> multistrap
>>
>> Ok
>>
>>> and there are probably more ways to split that into reviewable
>>> pieces
>>
>> Please questions and proposals.
>
> Yes, for v2.
Ok.
>
>>>
>>> So this review is not too verbose, we need smaller pieces.
>>>
>>> More inline
>>>
>>> Am Sun, 27 Aug 2017 18:13:34 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>
>>>> From: Frank Lenormand <lenormf@gmail.com>
>>>>
>>>> Adds the possibility to create an apt repository for newly built
>>>> packages and then use that repo for populating the target
>>>> filesystem.
>>>>
>>>> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>> meta-isar-bin/conf/layer.conf | 18 +++
>>>> meta-isar-bin/files/distributions.in | 3 +
>>>> meta-isar/conf/bblayers.conf.sample | 1 +
>>>> .../images/files/debian-configscript.sh | 8 ++
>>>> .../images/files/raspbian-configscript.sh | 8 ++
>>>> meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
>>>> meta/classes/dpkg.bbclass | 126
>>>> +++++++++++++++++++--
>>>> meta/classes/image.bbclass | 30 +++-- 8
>>>> files changed, 186 insertions(+), 16 deletions(-) create mode
>>>> 100644 meta-isar-bin/conf/layer.conf create mode 100644
>>>> meta-isar-bin/files/distributions.in
>>>>
>>>> diff --git a/meta-isar-bin/conf/layer.conf
>>>> b/meta-isar-bin/conf/layer.conf new file mode 100644
>>>> index 0000000..24534e1
>>>> --- /dev/null
>>>> +++ b/meta-isar-bin/conf/layer.conf
>>>> @@ -0,0 +1,18 @@
>>>> +# Enable package caching with '1'
>>>> +DEBCACHE_ENABLED ?= "0"
>>>> +
>>>> +# Codename of the repository created by the caching class
>>>> +DEBDISTRONAME = "isar"
>>>> +
>>>> +# Path to the caching repository
>>>> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>>> +
>>>> +# Path to the mount point of the repository within the target
>>>> rootfs, during +# population
>>>> +DEBCACHEMNT ?= "/opt/cache/apt"
>>>> +
>>>> +# Path to the databases used by `reprepro`
>>>> +DEBDBDIR ?= "${LAYERDIR}/db"
>>>> +
>>>> +# Path to the configuration files templates used by `reprepro`
>>>> +DEBFILESDIR ?= "${LAYERDIR}/files"
>>>> diff --git a/meta-isar-bin/files/distributions.in
>>>> b/meta-isar-bin/files/distributions.in new file mode 100644
>>>> index 0000000..59f4429
>>>> --- /dev/null
>>>> +++ b/meta-isar-bin/files/distributions.in
>>>> @@ -0,0 +1,3 @@
>>>> +Codename: {DISTRO_NAME}
>>>> +Architectures: i386 armhf source
>>>> +Components: main
>>>> diff --git a/meta-isar/conf/bblayers.conf.sample
>>>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
>>>> --- a/meta-isar/conf/bblayers.conf.sample
>>>> +++ b/meta-isar/conf/bblayers.conf.sample
>>>> @@ -8,6 +8,7 @@ BBFILES ?= ""
>>>> BBLAYERS ?= " \
>>>> ##ISARROOT##/meta \
>>>> ##ISARROOT##/meta-isar \
>>>> + ##ISARROOT##/meta-isar-bin \
>>>> "
>>>> BBLAYERS_NON_REMOVABLE ?= " \
>>>> ##ISARROOT##/meta \
>>>> diff --git
>>>> a/meta-isar/recipes-core/images/files/debian-configscript.sh
>>>> b/meta-isar/recipes-core/images/files/debian-configscript.sh index
>>>> 4ac37d0..b05babb 100644 ---
>>>> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++
>>>> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@
>>>> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
>>>> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
>>>> +readonly DEBCACHEMNT="$4"
>>>> +readonly DEBDISTRONAME="$5"
>>>>
>>>> cat >> /etc/default/locale << EOF
>>>> LANG=en_US.UTF-8
>>>> @@ -83,3 +85,9 @@ fi
>>>> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d"
>>>> ]; then rm -f $TARGET/usr/sbin/policy-rc.d
>>>> fi
>>>> +
>>>> +mkdir -p /etc/apt/sources.list.d/
>>>> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
>>>> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>>>> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>>>> +EOF
>>>> diff --git
>>>> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
>>>> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh
>>>> index 2454481..f995f4d 100644 ---
>>>> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++
>>>> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@
>>>> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
>>>> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
>>>> +readonly DEBCACHEMNT="$4"
>>>> +readonly DEBDISTRONAME="$5"
>>>>
>>>> cat >> /etc/default/locale << EOF
>>>> LANG=en_US.UTF-8
>>>> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
>>>> cat > /boot/config.txt << EOF
>>>> kernel=$KERNEL_IMAGE
>>>> EOF
>>>> +
>>>> +mkdir -p /etc/apt/sources.list.d/
>>>> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
>>>> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>>>> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
>>>> +EOF
>>>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>>>> b/meta-isar/recipes-core/images/isar-image-base.bb index
>>>> b679d97..85e6b27 100644 ---
>>>> a/meta-isar/recipes-core/images/isar-image-base.bb +++
>>>> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12
>>>> @@ do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
>>>> "${WORKDIR}/multistrap.conf" || true
>>>> # Configure root filesystem
>>>> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
>>>> ${BAUDRATE_TTY} \
>>>> - ${ROOTFS_DEV}
>>>> + sudo chroot ${S} /configscript.sh \
>>>> + ${MACHINE_SERIAL} \
>>>> + ${BAUDRATE_TTY} \
>>>> + ${ROOTFS_DEV} \
>>>> + ${DEBCACHEMNT} \
>>>> + ${DEBDISTRONAME}
>>>> sudo rm ${S}/configscript.sh
>>>> }
>>>>
>>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>>> index 360a95c..b1e201d 100644
>>>> --- a/meta/classes/dpkg.bbclass
>>>> +++ b/meta/classes/dpkg.bbclass
>>>> @@ -1,5 +1,5 @@
>>>> # This software is a part of ISAR.
>>>> -# Copyright (C) 2015-2016 ilbers GmbH
>>>> +# Copyright (C) 2015-2017 ilbers GmbH
>>>>
>>>> # Add dependency from buildchroot creation
>>>> DEPENDS += "buildchroot"
>>>> @@ -55,12 +55,124 @@ do_build() {
>>>> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
>>>> }
>>>>
>>>> -
>>>> # Install package to dedicated deploy directory
>>>> -do_install() {
>>>> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
>>>> +do_binary_deb_install() {
>>>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>>>> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
>>>> +
>>>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>>>> + # If `bitbake` is running for the first time, the cache
>>>> doesn't exist
>>>> + # yet and needs to be configured using a `distributions`
>>>> file.
>>>> + # A template stored in the layer directory is
>>>> pre-processed to
>>>> + # generate the configuration file, which is then placed in
>>>> the
>>>> + # appropriate directory.
>>>> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
>>>> + mkdir -p "${DIR_CACHE}/conf"
>>>> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
>>>> + "${DEBFILESDIR}/distributions.in" \
>>>> + > "${DIR_CACHE}/conf/distributions"
>>>> + fi
>>>
>>> This step should be in the image and not in every package. I guess
>>> it has a race with multiple packages running this init routine.
>>
>> Ok.
>>
>>>
>>>> + # Add binary and source packages to the deb cache
>>>> + # If the cache doesn't exist yet, it will be created using
>>>> the
>>>> + # `distributions` file generated above.
>>>> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while
>>>> read -r p; do
>>>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>>>> "${DIR_DB}" \
>>>> + -C main "include${p##*.}" "${DEBDISTRONAME}"
>>>> "${p}"
>>>> + done
>>>> + else
>>>> + # Deb caching is disabled, simply copy all binary packages
>>>> to the
>>>> + # deploy directory
>>>> + mkdir -p "${DEPLOY_DIR_DEB}"
>>>> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
>>>> + fi
>>>> }
>>>>
>>>> -addtask install after do_build
>>>> -do_install[dirs] = "${DEPLOY_DIR_DEB}"
>>>> -do_install[stamp-extra-info] = "${MACHINE}"
>>>> +addtask binary_deb_install after do_build
>>>> +do_binary_deb_install[stamp-extra-info] =
>>>> "${DISTRO}-${DISTRO_ARCH}"
>>>
>>> Here a task got renamed. This should be another patch ... one that i
>>> already tried to get in.
>>>
>>
>> This implemented in the next patch (to keep this patch in original
>> format), but yes, it should be done here.
>>
>>>> +
>>>> +# Deb caching lambda run during the parsing phase that checks
>>>> whether the +# current package has to be rebuilt, or taken from the
>>>> cache +python __anonymous () {
>>>> + if d.getVar("DEBCACHE_ENABLED", True) == "0":
>>>> + # Deb caching is disabled, do nothing
>>>> + return True
>>>> +
>>>> + PN = d.getVar("PN", True)
>>>> + PV = d.getVar("PV", True)
>>>> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True)
>>>> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
>>>> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True)
>>>> + DEBDBDIR = d.getVar("DEBDBDIR", True)
>>>> + DISTRO = d.getVar("DISTRO", True)
>>>> + path_cache = os.path.join(DEBCACHEDIR, DISTRO)
>>>> + path_databases = os.path.join(DEBDBDIR, DISTRO)
>>>> + path_distributions = os.path.join(path_cache, "conf",
>>>> "distributions") +
>>>> + # The distributions file is needed by `reprepro` to know what
>>>> types
>>>> + # of packages are supported, what the distribution name is,
>>>> etc.
>>>> + # If it doesn't exist, we have nothing in the cache, do
>>>> nothing.
>>>> + if not os.path.exists(path_distributions):
>>>> + return
>>>> +
>>>> + # Anonymous functions are run several times under different
>>>> contexts
>>>> + # during the parsing phase, which would let the code that
>>>> follows be run
>>>> + # as many times for the same package.
>>>> + # In order to guarantee that our subroutine only runs once per
>>>> package, we
>>>> + # use bitbake's "persist" API in order to have reliable
>>>> persistent storage
>>>> + # accross calls of the lambda (using a simple variable in the
>>>> class won't
>>>> + # work, as several contexts won't allow fetching its value).
>>>> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d)
>>>> + if PN in pd and pd[PN] == PV:
>>>> + return
>>>> +
>>>> + import subprocess
>>>> + try:
>>>> + # The databases used by `reprepro` are not stored within
>>>> the cache in
>>>> + # order to make versioning of only the files needed to use
>>>> the cache
>>>> + # as an official Debian repository simpler.
>>>> + # As such, if a developer uses a peer's cache to speed up
>>>> their build
>>>> + # time but have never run bitbake, the database will not
>>>> have been
>>>> + # created, so we regenerate them here.
>>>> + if not os.path.exists(path_databases) and
>>>> os.path.exists(path_cache):
>>>> + bb.note("Regenerating the cache databases...")
>>>> + subprocess.check_call([
>>>> + "reprepro",
>>>> + "--waitforlock", "3",
>>>> + "-b", path_cache,
>>>> + "--dbdir", path_databases,
>>>> + "export", DEBDISTRONAME,
>>>> + ])
>>>> +
>>>> + # Get a list of the versions of all the packages named
>>>> after the
>>>> + # current bitbake package, and check whether the current
>>>> package
>>>> + # version is returned.
>>>> + # As `reprepro` always returns zero with this particular
>>>> operation, we
>>>> + # have to use this workaround to check for a package in
>>>> the cache.
>>>> + package_version = subprocess.check_output([
>>>> + "reprepro",
>>>> + "--waitforlock", "3",
>>>> + "-b", path_cache,
>>>> + "--dbdir", path_databases,
>>>> + "-C", "main",
>>>> + "-A", DISTRO_ARCH,
>>>> + "--list-format", "${version}",
>>>> + "list", DEBDISTRONAME, PN,
>>>> + ])
>>>> + package_version = package_version.decode("utf-8")
>>>> + if package_version == PV:
>>>> + # The below list contains the names of all the tasks
>>>> are in charge
>>>> + # of building the package when the cache isn't enabled
>>>> or if the
>>>> + # package hasn't been placed in it already.
>>>> + # As all tasks are enabled by default, we prevent
>>>> their execution
>>>> + # by setting the `noexec` flag, which will prevent a
>>>> rebuild of
>>>> + # the package when it's cached.
>>>> + for task in ["fetch", "unpack", "build", "install"]:
>>>> + d.setVarFlag("do_{}".format(task), "noexec", "1")
>>>> +
>>>> + # Cache the results of this command so that subsequent
>>>> executions
>>>> + # of this anonymous functions don't run the same code
>>>> again.
>>>> + pd[PN] = PV
>>>> + except subprocess.CalledProcessError as e:
>>>> + bb.fatal("Unable to check for a candidate for package {0}
>>>> (errorcode: {1})".format(PN, e.returncode)) +}
>>>> diff --git a/meta/classes/image.bbclass
>>>> b/meta/classes/image.bbclass index a7f0d74..f42aa48 100644
>>>> --- a/meta/classes/image.bbclass
>>>> +++ b/meta/classes/image.bbclass
>>>> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
>>>>
>>>> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>>>
>>>> -# Install Debian packages, that were built from sources
>>>> +# Install Debian packages from the cache
>>>> do_populate() {
>>>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>>>> +
>>>> if [ -n "${IMAGE_INSTALL}" ]; then
>>>> - sudo mkdir -p ${S}/deb
>>>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>>>> + sudo mkdir -p "${S}/${DEBCACHEMNT}"
>>>> + sudo mount -o bind "${DIR_CACHE}"
>>>> "${S}/${DEBCACHEMNT}" +
>>>> + sudo chroot "${S}" apt-get update -y
>>>
>>> This will fail if you are behind a proxy. It makes a known issue
>>> worse.
>>>> + for package in ${IMAGE_INSTALL}; do
>>>> + sudo chroot "${S}" apt-get install -t
>>>> "${DEBDISTRONAME}" -y \
>>>> + --allow-unauthenticated "${package}"
>>>> + done
>>>> +
>>>> + sudo umount "${S}/${DEBCACHEMNT}"
>>>
>>> A whole lot of sudo. I do not mind but am still wondering ...
>>
>> Please questions and proposals instead of negative tone.
>
> That was maybe slightly negative. Why? Because that patch was sent at a
> time where the policy was still "do not add more sudos". And the patch
> violates more of the groundrules we have agreed on.
That is actually not true, because the policy still is "do not add more
sudo if you have no use-case for them".
I'd like to cover this topic again to try to explain the issue. The
thread was started about your comment:
> Some security enhancing packages can cause our initrd to be not
> readable by a normal user. So we need to copy with sudo.
> Also regular cp would destroy ownership and other attributes of files,
> possibly creating problems in the future.
and in the patch you added three sudo instead of single one for initrd.
No problem with sudo for initrd - you provided use-case. But "possibly
creating problems in the future" doesn't provide use-case for the
remaining two entries.
> You posting the patch signals that our groundrules seem to apply
> to !ilbers-contributions only.
>
>>> Can we replace the bind-mount with a symlink? If anything between
>>> the mount and umount fails and you need to run again ... will the
>>> mount fail if the old one is still there?
>>
>> If we switch to multistrap-based populating, I think this will go.
>
> I know, but we have not decided that yet.
>
> Henning
>
>>>
>>>> + else
>>>> + sudo mkdir -p ${S}/deb
>>>>
>>>> - for p in ${IMAGE_INSTALL}; do
>>>> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
>>>> - done
>>>> + for p in ${IMAGE_INSTALL}; do
>>>> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb'
>>>> -exec \
>>>> + sudo cp '{}' "${S}/deb/" \;
>>>> + done
>>>>
>>>> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb
>>>> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb
>>>>
>>>> - sudo rm -rf ${S}/deb
>>>> + sudo rm -rf ${S}/deb
>>>> + fi
>>>> fi
>>>> }
>>>>
>>>
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 5/6] classes/dpkg: Properly update packages in the cache
2017-08-29 7:57 ` Henning Schild
@ 2017-08-29 11:26 ` Jan Kiszka
0 siblings, 0 replies; 30+ messages in thread
From: Jan Kiszka @ 2017-08-29 11:26 UTC (permalink / raw)
To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users
On 2017-08-29 09:57, [ext] Henning Schild wrote:
> Am Tue, 29 Aug 2017 10:20:41 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>>> This is another fixup of 1/6, fold it in.
>>>
>>
>> It's a good question if we want to keep development history. Because
>> these patches reflect the real history of binary caching development.
>
> To me the answer is clear. If we find issues in a patch during review,
> the patch should be fixed before getting merged.
> If we do not follow that policy you might as well merge all patches
> without review.
ACK. This is how many many other projects work as well.
Jan
--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 6/6] doc/technical_overview: Describe binary cache
2017-08-28 15:36 ` Henning Schild
2017-08-29 7:29 ` Alexander Smirnov
@ 2017-08-29 11:29 ` Jan Kiszka
1 sibling, 0 replies; 30+ messages in thread
From: Jan Kiszka @ 2017-08-29 11:29 UTC (permalink / raw)
To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users
On 2017-08-28 17:36, [ext] Henning Schild wrote:
> This should be done along the way in every single patch, not afterwards.
>
I wouldn't be too strict with documentation, though. Generally, updating
the docs in one step after a longer series of changes is acceptable as
well as long as it happens in the same series. Keeping the code
functionality intact in each commit is mandatory (with extremely few
exceptions).
Jan
--
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 3/6] classes/dpkg: Split install for cache
2017-08-29 7:18 ` Alexander Smirnov
@ 2017-08-30 8:54 ` Claudius Heine
0 siblings, 0 replies; 30+ messages in thread
From: Claudius Heine @ 2017-08-30 8:54 UTC (permalink / raw)
To: Alexander Smirnov, isar-users
Hi,
On 08/29/2017 09:18 AM, Alexander Smirnov wrote:
>
>
> On 08/28/2017 11:00 AM, Claudius Heine wrote:
>> Hi,
>>
>> On 08/27/2017 05:13 PM, Alexander Smirnov wrote:
>>> Split install function into two parts:
>>> - install_to_cache: if caching is enabled
>>> - install_to_deploy: if caching is disabled
>>>
>>> This patch brings flexibility to the implementation and makes it
>>> possible
>>> to move all the caching implementation code to dedicated class. The
>>> magic
>>> behavior depending on the value of DEBCACHE_ENABLED is now
>>> transparent in
>>> the bitbake pipeline (can be inspected via "bitbake -g").
>>>
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>> meta/classes/dpkg.bbclass | 66
>>> ++++++++++++++++++++++++++--------------------
>>> meta/classes/image.bbclass | 2 +-
>>> 2 files changed, 38 insertions(+), 30 deletions(-)
>>>
>>> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
>>> index b1e201d..118ba2f 100644
>>> --- a/meta/classes/dpkg.bbclass
>>> +++ b/meta/classes/dpkg.bbclass
>>> @@ -56,46 +56,53 @@ do_build() {
>>> }
>>> # Install package to dedicated deploy directory
>>> -do_binary_deb_install() {
>>> +do_install_to_deploy() {
>>> + # Deb caching is disabled, simply copy all binary packages to
>>> the deploy
>>> + # directory
>>> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
>>> +}
>>> +
>>> +addtask install_to_deploy after do_build
>>> +do_install_to_deploy[dirs] = "${DEPLOY_DIR_DEB}"
>>> +do_install_to_deploy[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>> +do_install_to_deploy[noexec] = "1"
>>> +
>>> +# Install package to dedicated apt repo
>>> +do_install_to_cache() {
>>> readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>>> readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
>>> - if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>>> - # If `bitbake` is running for the first time, the cache
>>> doesn't exist
>>> - # yet and needs to be configured using a `distributions` file.
>>> - # A template stored in the layer directory is pre-processed to
>>> - # generate the configuration file, which is then placed in the
>>> - # appropriate directory.
>>> - if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
>>> - mkdir -p "${DIR_CACHE}/conf"
>>> - sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
>>> - "${DEBFILESDIR}/distributions.in" \
>>> - > "${DIR_CACHE}/conf/distributions"
>>> - fi
>>> -
>>> - # Add binary and source packages to the deb cache
>>> - # If the cache doesn't exist yet, it will be created using the
>>> - # `distributions` file generated above.
>>> - ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
>>> -r p; do
>>> - reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>>> "${DIR_DB}" \
>>> - -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
>>> - done
>>> - else
>>> - # Deb caching is disabled, simply copy all binary packages
>>> to the
>>> - # deploy directory
>>> - mkdir -p "${DEPLOY_DIR_DEB}"
>>> - install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
>>> + # If `bitbake` is running for the first time, the cache doesn't
>>> exist
>>> + # yet and needs to be configured using a `distributions` file.
>>> + # A template stored in the layer directory is pre-processed to
>>> + # generate the configuration file, which is then placed in the
>>> + # appropriate directory.
>>> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
>>> + mkdir -p "${DIR_CACHE}/conf"
>>> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
>>> + "${DEBFILESDIR}/distributions.in" \
>>> + > "${DIR_CACHE}/conf/distributions"
>>> fi
>>> +
>>> + # Add binary and source packages to the deb cache
>>> + # If the cache doesn't exist yet, it will be created using the
>>> + # `distributions` file generated above.
>>> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read -r
>>> p; do
>>> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
>>> "${DIR_DB}" \
>>> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
>>> + done
>>> }
>>> -addtask binary_deb_install after do_build
>>> -do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>> +addtask install_to_cache after do_build
>>
>> What was the reason why we put stuff after the 'build' task? In OE the
>> 'build' task is the default task triggering everything necessary for a
>> recipe. In Isar that changed. Why?
>
> Actually AFAIK do_build is not OE tasks, it's default bitbake task.
> Install is after do_build, because in Isar do_build is used to build the
> packages, so it's not empty.
You are right, its a bitbake task. Since I have much more experience
with OE than with just bitbake or isar, I referenced my question to the
experience I had using OE. I cannot remember seeing tasks that are added
after the do_build task in OE and I still find that very strange. AFAIK
the 'build' task in bitbake and OE has a similar meaning to the 'all'
target in make files. The idea to add statements to the 'all' target or
letting other targets depend on the 'all' target sounds very odd to me.
And that is what is done here in isar. Just very strange and unusual for
my eyes and I don't see the advantage of this.
I would like to hear the reason for this decision. Why have you added
statements to the 'build' task and why are other tasks depend on it?
>>
>>> +do_install_to_cache[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
>>> +do_install_to_cache[noexec] = "1"
>>> # Deb caching lambda run during the parsing phase that checks
>>> whether the
>>> # current package has to be rebuilt, or taken from the cache
>>> python __anonymous () {
>>> if d.getVar("DEBCACHE_ENABLED", True) == "0":
>>> - # Deb caching is disabled, do nothing
>>> + # Deb caching is disabled
>>> + d.delVarFlag('do_install_to_deploy', 'noexec')
>>> return True
>>> PN = d.getVar("PN", True)
>>> @@ -108,6 +115,7 @@ python __anonymous () {
>>> path_cache = os.path.join(DEBCACHEDIR, DISTRO)
>>> path_databases = os.path.join(DEBDBDIR, DISTRO)
>>> path_distributions = os.path.join(path_cache, "conf",
>>> "distributions")
>>> + d.delVarFlag('do_install_to_cache', 'noexec')
>>> # The distributions file is needed by `reprepro` to know what
>>> types
>>> # of packages are supported, what the distribution name is, etc.
>>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>>> index c2ff453..6b1b5eb 100644
>>> --- a/meta/classes/image.bbclass
>>> +++ b/meta/classes/image.bbclass
>>> @@ -46,4 +46,4 @@ do_populate() {
>>> }
>>> addtask populate before do_build
>>> -do_populate[deptask] = "do_install"
>>> +do_populate[deptask] = "do_install_to_cache do_install_to_deploy"
>>
>> I would rather still have a, maybe empty, 'do_install' step that just
>> depends on the 'do_install_to_cache' and 'do_install_to_deploy' tasks.
>> I don't see why ever small change in one class should change the
>> complete pipeline for every recipe.
>
> It's a bit difficult question. In general meaning when building
> something from sources, the install means a storing of build artifacts
> to some specific location. So in this case, we just copy binary deb
> package to repo, and there is nothing common with installation of
> artifacts.
>
> Also Isar has in roadmap cross-compilation, so the pipeline will
> probably look like:
>
> fetch - unpack - compile - install - install_to_cache
>
> so in this case the install task probably should go before
> install_to_cache.
>
> So probably it makes sense to avoid _install_ in this task, for example
> use do_populate_deb_cache, what do you think?
Something like: 'populate_repo' sounds much better for this case. I
would try to have more abstract names at first and implement them with
more specific names like 'populate_repo_deb'. In OE there already is a
'package' and additional sub tasks like 'package_write_deb'.
So a general pipeline for normal recipes like this:
fetch, unpack, configure, compile, install, package, populate_repo
Of course image recipes would have a slightly changed pipeline.
I think the discussion about the build pipeline in isar is very
important. Maybe we should make an extra thread about this.
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-28 15:18 ` Henning Schild
2017-08-29 6:40 ` Alexander Smirnov
@ 2017-08-31 10:55 ` Claudius Heine
2017-08-31 11:20 ` Henning Schild
1 sibling, 1 reply; 30+ messages in thread
From: Claudius Heine @ 2017-08-31 10:55 UTC (permalink / raw)
To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users, Frank Lenormand
Hi,
On 08/28/2017 05:18 PM, [ext] Henning Schild wrote:
> - i think this task should go away and the repo should be given to
> multistrap
Yes please! Just found another issue why installing all packages that
are in one directory is bad. The directory path (DEPLOY_DIR_DEB) is not
and cannot be image specific, so building multiple images in the same
environment causes packages to be installed just because they are part
of an image that was build previously.
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-31 10:55 ` Claudius Heine
@ 2017-08-31 11:20 ` Henning Schild
2017-08-31 12:08 ` Claudius Heine
0 siblings, 1 reply; 30+ messages in thread
From: Henning Schild @ 2017-08-31 11:20 UTC (permalink / raw)
To: Claudius Heine; +Cc: Alexander Smirnov, isar-users, Frank Lenormand
Am Thu, 31 Aug 2017 12:55:52 +0200
schrieb Claudius Heine <claudius.heine.ext@siemens.com>:
> Hi,
>
> On 08/28/2017 05:18 PM, [ext] Henning Schild wrote:
> > - i think this task should go away and the repo should be given to
> > multistrap
>
> Yes please! Just found another issue why installing all packages that
> are in one directory is bad. The directory path (DEPLOY_DIR_DEB) is
> not and cannot be image specific, so building multiple images in the
> same environment causes packages to be installed just because they
> are part of an image that was build previously.
This comment does not address the point of feeding the repo into
multistrap instead of doing do_populate.
Are you sure you are using the do_populate as in master?
> for p in ${IMAGE_INSTALL}; do
> sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
> done
> sudo chroot ${S} /usr/bin/dpkg -i -R /deb
This looks like not all are installed. But only the mentioned ones.
It also looks like you must never have /deb in your rootfs but that is
another issue that will get fixed when we get rid of do_populate.
Henning
> Cheers,
> Claudius
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-31 11:20 ` Henning Schild
@ 2017-08-31 12:08 ` Claudius Heine
0 siblings, 0 replies; 30+ messages in thread
From: Claudius Heine @ 2017-08-31 12:08 UTC (permalink / raw)
To: Henning Schild; +Cc: Alexander Smirnov, isar-users, Frank Lenormand
Hi,
On 08/31/2017 01:20 PM, Henning Schild wrote:
> Am Thu, 31 Aug 2017 12:55:52 +0200
> schrieb Claudius Heine <claudius.heine.ext@siemens.com>:
>
>> Hi,
>>
>> On 08/28/2017 05:18 PM, [ext] Henning Schild wrote:
>>> - i think this task should go away and the repo should be given to
>>> multistrap
>>
>> Yes please! Just found another issue why installing all packages that
>> are in one directory is bad. The directory path (DEPLOY_DIR_DEB) is
>> not and cannot be image specific, so building multiple images in the
>> same environment causes packages to be installed just because they
>> are part of an image that was build previously.
>
> This comment does not address the point of feeding the repo into
> multistrap instead of doing do_populate.
>
> Are you sure you are using the do_populate as in master?
Nope, sorry. I used modified code here that handles package dependencies
and this is the cause for this issue. But I would love to get rid of
these kind of hacks this way!
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
2017-08-28 15:18 ` Henning Schild
@ 2017-09-06 14:21 ` Henning Schild
2017-09-07 11:13 ` Claudius Heine
1 sibling, 1 reply; 30+ messages in thread
From: Henning Schild @ 2017-09-06 14:21 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users, Frank Lenormand
Am Sun, 27 Aug 2017 18:13:34 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> From: Frank Lenormand <lenormf@gmail.com>
>
> Adds the possibility to create an apt repository for newly built
> packages and then use that repo for populating the target filesystem.
>
> Signed-off-by: Frank Lenormand <lenormf@gmail.com>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar-bin/conf/layer.conf | 18 +++
> meta-isar-bin/files/distributions.in | 3 +
> meta-isar/conf/bblayers.conf.sample | 1 +
> .../images/files/debian-configscript.sh | 8 ++
> .../images/files/raspbian-configscript.sh | 8 ++
> meta-isar/recipes-core/images/isar-image-base.bb | 8 +-
> meta/classes/dpkg.bbclass | 126
> +++++++++++++++++++--
> meta/classes/image.bbclass | 30 +++-- 8
> files changed, 186 insertions(+), 16 deletions(-) create mode 100644
> meta-isar-bin/conf/layer.conf create mode 100644
> meta-isar-bin/files/distributions.in
>
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf new file mode 100644
> index 0000000..24534e1
> --- /dev/null
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -0,0 +1,18 @@
> +# Enable package caching with '1'
> +DEBCACHE_ENABLED ?= "0"
> +
> +# Codename of the repository created by the caching class
> +DEBDISTRONAME = "isar"
> +
> +# Path to the caching repository
> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> +
> +# Path to the mount point of the repository within the target
> rootfs, during +# population
> +DEBCACHEMNT ?= "/opt/cache/apt"
> +
> +# Path to the databases used by `reprepro`
> +DEBDBDIR ?= "${LAYERDIR}/db"
> +
> +# Path to the configuration files templates used by `reprepro`
> +DEBFILESDIR ?= "${LAYERDIR}/files"
> diff --git a/meta-isar-bin/files/distributions.in
> b/meta-isar-bin/files/distributions.in new file mode 100644
> index 0000000..59f4429
> --- /dev/null
> +++ b/meta-isar-bin/files/distributions.in
> @@ -0,0 +1,3 @@
> +Codename: {DISTRO_NAME}
> +Architectures: i386 armhf source
> +Components: main
> diff --git a/meta-isar/conf/bblayers.conf.sample
> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> --- a/meta-isar/conf/bblayers.conf.sample
> +++ b/meta-isar/conf/bblayers.conf.sample
> @@ -8,6 +8,7 @@ BBFILES ?= ""
> BBLAYERS ?= " \
> ##ISARROOT##/meta \
> ##ISARROOT##/meta-isar \
> + ##ISARROOT##/meta-isar-bin \
> "
> BBLAYERS_NON_REMOVABLE ?= " \
> ##ISARROOT##/meta \
> diff --git
> a/meta-isar/recipes-core/images/files/debian-configscript.sh
> b/meta-isar/recipes-core/images/files/debian-configscript.sh index
> 4ac37d0..b05babb 100644 ---
> a/meta-isar/recipes-core/images/files/debian-configscript.sh +++
> b/meta-isar/recipes-core/images/files/debian-configscript.sh @@ -8,6
> +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly BAUDRATE_TTY="$2"
> readonly ROOTFS_DEV="$3"
> +readonly DEBCACHEMNT="$4"
> +readonly DEBDISTRONAME="$5"
>
> cat >> /etc/default/locale << EOF
> LANG=en_US.UTF-8
> @@ -83,3 +85,9 @@ fi
> if [ -x "$TARGET/sbin/init" -a -x "$TARGET/usr/sbin/policy-rc.d" ];
> then rm -f $TARGET/usr/sbin/policy-rc.d
> fi
> +
> +mkdir -p /etc/apt/sources.list.d/
> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +EOF
> diff --git
> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh
> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh index
> 2454481..f995f4d 100644 ---
> a/meta-isar/recipes-core/images/files/raspbian-configscript.sh +++
> b/meta-isar/recipes-core/images/files/raspbian-configscript.sh @@
> -8,6 +8,8 @@ set -e readonly MACHINE_SERIAL="$1" readonly
> BAUDRATE_TTY="$2" readonly ROOTFS_DEV="$3"
> +readonly DEBCACHEMNT="$4"
> +readonly DEBDISTRONAME="$5"
>
> cat >> /etc/default/locale << EOF
> LANG=en_US.UTF-8
> @@ -89,3 +91,9 @@ KERNEL_IMAGE=`ls /boot | grep vmlinuz`
> cat > /boot/config.txt << EOF
> kernel=$KERNEL_IMAGE
> EOF
> +
> +mkdir -p /etc/apt/sources.list.d/
> +cat <<EOF >/etc/apt/sources.list.d/${DEBDISTRONAME}.list
> +deb file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +deb-src file:${DEBCACHEMNT}/ ${DEBDISTRONAME} main
> +EOF
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> b679d97..85e6b27 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -49,8 +49,12 @@
> do_rootfs() { sudo multistrap -a ${DISTRO_ARCH} -d "${S}" -f
> "${WORKDIR}/multistrap.conf" || true
> # Configure root filesystem
> - sudo chroot ${S} /configscript.sh ${MACHINE_SERIAL}
> ${BAUDRATE_TTY} \
> - ${ROOTFS_DEV}
> + sudo chroot ${S} /configscript.sh \
> + ${MACHINE_SERIAL} \
> + ${BAUDRATE_TTY} \
> + ${ROOTFS_DEV} \
> + ${DEBCACHEMNT} \
> + ${DEBDISTRONAME}
> sudo rm ${S}/configscript.sh
> }
>
> diff --git a/meta/classes/dpkg.bbclass b/meta/classes/dpkg.bbclass
> index 360a95c..b1e201d 100644
> --- a/meta/classes/dpkg.bbclass
> +++ b/meta/classes/dpkg.bbclass
> @@ -1,5 +1,5 @@
> # This software is a part of ISAR.
> -# Copyright (C) 2015-2016 ilbers GmbH
> +# Copyright (C) 2015-2017 ilbers GmbH
>
> # Add dependency from buildchroot creation
> DEPENDS += "buildchroot"
> @@ -55,12 +55,124 @@ do_build() {
> sudo chroot ${BUILDCHROOT_DIR} /build.sh ${PP}/${SRC_DIR}
> }
>
> -
> # Install package to dedicated deploy directory
> -do_install() {
> - install -m 644 ${BUILDROOT}/*.deb ${DEPLOY_DIR_DEB}/
> +do_binary_deb_install() {
> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> + readonly DIR_DB="${DEBDBDIR}/${DISTRO}"
> +
> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> + # If `bitbake` is running for the first time, the cache
> doesn't exist
> + # yet and needs to be configured using a `distributions`
> file.
> + # A template stored in the layer directory is pre-processed
> to
> + # generate the configuration file, which is then placed in
> the
> + # appropriate directory.
> + if [ ! -e "${DIR_CACHE}/conf/distributions" ]; then
> + mkdir -p "${DIR_CACHE}/conf"
> + sed -e "s#{DISTRO_NAME}#${DEBDISTRONAME}#g" \
> + "${DEBFILESDIR}/distributions.in" \
> + > "${DIR_CACHE}/conf/distributions"
> + fi
> +
> + # Add binary and source packages to the deb cache
> + # If the cache doesn't exist yet, it will be created using
> the
> + # `distributions` file generated above.
> + ls -1 "${BUILDROOT}"/*.deb "${BUILDROOT}"/*.dsc | while read
> -r p; do
> + reprepro --waitforlock 3 -b "${DIR_CACHE}" --dbdir
> "${DIR_DB}" \
> + -C main "include${p##*.}" "${DEBDISTRONAME}" "${p}"
> + done
> + else
> + # Deb caching is disabled, simply copy all binary packages
> to the
> + # deploy directory
> + mkdir -p "${DEPLOY_DIR_DEB}"
> + install -m 644 "${BUILDROOT}"/*.deb "${DEPLOY_DIR_DEB}/"
> + fi
> }
>
> -addtask install after do_build
> -do_install[dirs] = "${DEPLOY_DIR_DEB}"
> -do_install[stamp-extra-info] = "${MACHINE}"
> +addtask binary_deb_install after do_build
> +do_binary_deb_install[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
> +
> +# Deb caching lambda run during the parsing phase that checks
> whether the +# current package has to be rebuilt, or taken from the
> cache +python __anonymous () {
> + if d.getVar("DEBCACHE_ENABLED", True) == "0":
> + # Deb caching is disabled, do nothing
> + return True
> +
> + PN = d.getVar("PN", True)
> + PV = d.getVar("PV", True)
> + DISTRO_ARCH = d.getVar("DISTRO_ARCH", True)
> + DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
> + DEBDISTRONAME = d.getVar("DEBDISTRONAME", True)
> + DEBDBDIR = d.getVar("DEBDBDIR", True)
> + DISTRO = d.getVar("DISTRO", True)
> + path_cache = os.path.join(DEBCACHEDIR, DISTRO)
> + path_databases = os.path.join(DEBDBDIR, DISTRO)
> + path_distributions = os.path.join(path_cache, "conf",
> "distributions") +
> + # The distributions file is needed by `reprepro` to know what
> types
> + # of packages are supported, what the distribution name is, etc.
> + # If it doesn't exist, we have nothing in the cache, do nothing.
> + if not os.path.exists(path_distributions):
> + return
> +
> + # Anonymous functions are run several times under different
> contexts
> + # during the parsing phase, which would let the code that
> follows be run
> + # as many times for the same package.
> + # In order to guarantee that our subroutine only runs once per
> package, we
> + # use bitbake's "persist" API in order to have reliable
> persistent storage
> + # accross calls of the lambda (using a simple variable in the
> class won't
> + # work, as several contexts won't allow fetching its value).
> + pd = bb.persist_data.persist("DEBCACHE_PACKAGES", d)
> + if PN in pd and pd[PN] == PV:
> + return
> +
> + import subprocess
> + try:
> + # The databases used by `reprepro` are not stored within the
> cache in
> + # order to make versioning of only the files needed to use
> the cache
> + # as an official Debian repository simpler.
> + # As such, if a developer uses a peer's cache to speed up
> their build
> + # time but have never run bitbake, the database will not
> have been
> + # created, so we regenerate them here.
> + if not os.path.exists(path_databases) and
> os.path.exists(path_cache):
> + bb.note("Regenerating the cache databases...")
> + subprocess.check_call([
> + "reprepro",
> + "--waitforlock", "3",
> + "-b", path_cache,
> + "--dbdir", path_databases,
> + "export", DEBDISTRONAME,
> + ])
> +
> + # Get a list of the versions of all the packages named after
> the
> + # current bitbake package, and check whether the current
> package
> + # version is returned.
> + # As `reprepro` always returns zero with this particular
> operation, we
> + # have to use this workaround to check for a package in the
> cache.
> + package_version = subprocess.check_output([
> + "reprepro",
> + "--waitforlock", "3",
> + "-b", path_cache,
> + "--dbdir", path_databases,
> + "-C", "main",
> + "-A", DISTRO_ARCH,
> + "--list-format", "${version}",
> + "list", DEBDISTRONAME, PN,
> + ])
> + package_version = package_version.decode("utf-8")
> + if package_version == PV:
> + # The below list contains the names of all the tasks are
> in charge
> + # of building the package when the cache isn't enabled
> or if the
> + # package hasn't been placed in it already.
> + # As all tasks are enabled by default, we prevent their
> execution
> + # by setting the `noexec` flag, which will prevent a
> rebuild of
> + # the package when it's cached.
> + for task in ["fetch", "unpack", "build", "install"]:
> + d.setVarFlag("do_{}".format(task), "noexec", "1")
> +
> + # Cache the results of this command so that subsequent
> executions
> + # of this anonymous functions don't run the same code
> again.
> + pd[PN] = PV
> + except subprocess.CalledProcessError as e:
> + bb.fatal("Unable to check for a candidate for package {0}
> (errorcode: {1})".format(PN, e.returncode)) +}
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index a7f0d74..f42aa48 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
>
> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>
> -# Install Debian packages, that were built from sources
> +# Install Debian packages from the cache
> do_populate() {
> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
> +
> if [ -n "${IMAGE_INSTALL}" ]; then
> - sudo mkdir -p ${S}/deb
> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
> + sudo mkdir -p "${S}/${DEBCACHEMNT}"
> + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
> +
> + sudo chroot "${S}" apt-get update -y
> + for package in ${IMAGE_INSTALL}; do
> + sudo chroot "${S}" apt-get install -t
> "${DEBDISTRONAME}" -y \
> + --allow-unauthenticated "${package}"
With do_populate using apt-get we now how to deal with package-name
collisions. We could have the case where our repo contains a package
that already exists in upstream debian.
We will have to decide how to deal with them if they occur. My
suggestions/first idea would be:
- a recipe that creates a .deb with a taken package name fails by
default -EBUSY
- a recipe can set a variable to force itself in "OVERRIDE_DEBIAN =
True"
- a recipe that overrides the upstream package has to "win", we need
a proper apt Pin-Priority
Henning
> + done
> +
> + sudo umount "${S}/${DEBCACHEMNT}"
> + else
> + sudo mkdir -p ${S}/deb
>
> - for p in ${IMAGE_INSTALL}; do
> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${S}/deb
> - done
> + for p in ${IMAGE_INSTALL}; do
> + find "${DEPLOY_DIR_DEB}" -type f -name '*.deb' -exec
> \
> + sudo cp '{}' "${S}/deb/" \;
> + done
>
> - sudo chroot ${S} /usr/bin/dpkg -i -R /deb
> + sudo chroot ${S} /usr/bin/dpkg -i -R /deb
>
> - sudo rm -rf ${S}/deb
> + sudo rm -rf ${S}/deb
> + fi
> fi
> }
>
^ permalink raw reply [flat|nested] 30+ messages in thread
* Re: [PATCH 1/6] meta-isar-bin: Enable caching of deb packages
2017-09-06 14:21 ` Henning Schild
@ 2017-09-07 11:13 ` Claudius Heine
0 siblings, 0 replies; 30+ messages in thread
From: Claudius Heine @ 2017-09-07 11:13 UTC (permalink / raw)
To: [ext] Henning Schild, Alexander Smirnov; +Cc: isar-users, Frank Lenormand
Hi,
On 09/06/2017 04:21 PM, [ext] Henning Schild wrote:
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -11,18 +11,34 @@ inherit ${IMAGE_TYPE}
>>
>> do_populate[stamp-extra-info] = "${MACHINE}-${DISTRO}"
>>
>> -# Install Debian packages, that were built from sources
>> +# Install Debian packages from the cache
>> do_populate() {
>> + readonly DIR_CACHE="${DEBCACHEDIR}/${DISTRO}"
>> +
>> if [ -n "${IMAGE_INSTALL}" ]; then
>> - sudo mkdir -p ${S}/deb
>> + if [ "${DEBCACHE_ENABLED}" != "0" ]; then
>> + sudo mkdir -p "${S}/${DEBCACHEMNT}"
>> + sudo mount -o bind "${DIR_CACHE}" "${S}/${DEBCACHEMNT}"
>> +
>> + sudo chroot "${S}" apt-get update -y
>> + for package in ${IMAGE_INSTALL}; do
>> + sudo chroot "${S}" apt-get install -t
>> "${DEBDISTRONAME}" -y \
>> + --allow-unauthenticated "${package}"
>
> With do_populate using apt-get we now how to deal with package-name
> collisions. We could have the case where our repo contains a package
> that already exists in upstream debian.
>
> We will have to decide how to deal with them if they occur. My
> suggestions/first idea would be:
> - a recipe that creates a .deb with a taken package name fails by
> default -EBUSY
> - a recipe can set a variable to force itself in "OVERRIDE_DEBIAN =
> True"
> - a recipe that overrides the upstream package has to "win", we need
> a proper apt Pin-Priority
If all packages that are created by isar have for instance 'isar' in the
version, then it should be possible to specify this version explicitly
when installing them from the repository. I think pining is only
necessary if upgrading packages on the target is a required feature.
One option for this would be that another meta package (maybe created in
the image recipe) has explicit version dependencies on all packages
created by isar. This way apt would not upgrade those packages because
it would break the version dependency.
Cheers,
Claudius
--
DENX Software Engineering GmbH, Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de
^ permalink raw reply [flat|nested] 30+ messages in thread
end of thread, other threads:[~2017-09-07 11:13 UTC | newest]
Thread overview: 30+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-08-27 15:13 [PATCH 0/6] Isar apt cache implementation Alexander Smirnov
2017-08-27 15:13 ` [PATCH 1/6] meta-isar-bin: Enable caching of deb packages Alexander Smirnov
2017-08-28 15:18 ` Henning Schild
2017-08-29 6:40 ` Alexander Smirnov
2017-08-29 7:51 ` Henning Schild
2017-08-29 8:20 ` Alexander Smirnov
2017-08-31 10:55 ` Claudius Heine
2017-08-31 11:20 ` Henning Schild
2017-08-31 12:08 ` Claudius Heine
2017-09-06 14:21 ` Henning Schild
2017-09-07 11:13 ` Claudius Heine
2017-08-27 15:13 ` [PATCH 2/6] classes/image: Provide /dev/null for Stretch apt Alexander Smirnov
2017-08-28 15:20 ` Henning Schild
2017-08-28 15:26 ` Henning Schild
2017-08-27 15:13 ` [PATCH 3/6] classes/dpkg: Split install for cache Alexander Smirnov
2017-08-28 8:00 ` Claudius Heine
2017-08-29 7:18 ` Alexander Smirnov
2017-08-30 8:54 ` Claudius Heine
2017-08-28 15:30 ` Henning Schild
2017-08-27 15:13 ` [PATCH 4/6] meta-isar-bin: Enable apt repo generation for amd64 Alexander Smirnov
2017-08-27 15:13 ` [PATCH 5/6] classes/dpkg: Properly update packages in the cache Alexander Smirnov
2017-08-28 15:32 ` Henning Schild
2017-08-29 7:20 ` Alexander Smirnov
2017-08-29 7:57 ` Henning Schild
2017-08-29 11:26 ` Jan Kiszka
2017-08-27 15:13 ` [PATCH 6/6] doc/technical_overview: Describe binary cache Alexander Smirnov
2017-08-28 15:36 ` Henning Schild
2017-08-29 7:29 ` Alexander Smirnov
2017-08-29 8:06 ` Henning Schild
2017-08-29 11:29 ` Jan Kiszka
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox