* [PATCH v3 0/4] Basic binary cache implementation
@ 2017-09-25 9:59 Alexander Smirnov
2017-09-25 9:59 ` [PATCH v3 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov
` (3 more replies)
0 siblings, 4 replies; 10+ messages in thread
From: Alexander Smirnov @ 2017-09-25 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Hello everybody,
this series introduces basic binary caching for Isar.
There is a new layer: meta-isar-bin which is intended to be the binary
cache. All the packages that are built in Isar are stored in this
cache using reprepro.
Having the separate layer makes possible to manage this cache separately
from the Isar build tree. So once you have built your packages, you could
re-use this cache for further builds.
Also this cache repo is now used to generate image filesystem via
multistrap.
Changes since v2:
- Droped anonymous functions
- Added number of attempts to access apt cache before fail
Changes since v1:
- Total refactoring
- Split huge patches into small pieces
- Avoid races
With best regards,
Alex
8<--
Alexander Smirnov (4):
meta-isar-bin: Add reprepro configs
meta-isar-bin: Generate cache repos
meta-isar-bin: Populate cache
meta-isar-bin: Install packages via multistrap
meta-isar-bin/conf/layer.conf | 14 ++++++++
meta-isar-bin/files/distributions.in | 3 ++
meta-isar/conf/bblayers.conf.sample | 1 +
meta-isar/conf/local.conf.sample | 4 +++
.../recipes-core/images/files/multistrap.conf.in | 9 ++++-
meta-isar/recipes-core/images/isar-image-base.bb | 5 ++-
meta/classes/ext4-img.bbclass | 2 +-
meta/classes/image.bbclass | 41 ++++++++++++++++++----
8 files changed, 69 insertions(+), 10 deletions(-)
create mode 100644 meta-isar-bin/conf/layer.conf
create mode 100644 meta-isar-bin/files/distributions.in
--
2.1.4
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 1/4] meta-isar-bin: Add reprepro configs
2017-09-25 9:59 [PATCH v3 0/4] Basic binary cache implementation Alexander Smirnov
@ 2017-09-25 9:59 ` Alexander Smirnov
2017-09-25 11:25 ` Henning Schild
2017-09-25 9:59 ` [PATCH v3 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov
` (2 subsequent siblings)
3 siblings, 1 reply; 10+ messages in thread
From: Alexander Smirnov @ 2017-09-25 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Add template for reprepro configuration file and function that
generates final config for specific deistro.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar-bin/conf/layer.conf | 11 +++++++++++
meta-isar-bin/files/distributions.in | 3 +++
meta-isar/conf/bblayers.conf.sample | 1 +
meta/classes/image.bbclass | 13 +++++++++++++
4 files changed, 28 insertions(+)
create mode 100644 meta-isar-bin/conf/layer.conf
create mode 100644 meta-isar-bin/files/distributions.in
diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
new file mode 100644
index 0000000..3518184
--- /dev/null
+++ b/meta-isar-bin/conf/layer.conf
@@ -0,0 +1,11 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017 ilbers GmbH
+
+# Codename of the repository created by the caching class
+DEBDISTRONAME = "isar"
+
+# Path to the caching repository
+DEBCACHEDIR ?= "${LAYERDIR}/apt"
+
+# Path to the configuration files templates used by `reprepro`
+DEBFILESDIR ?= "${LAYERDIR}/files"
diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta-isar-bin/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta-isar/conf/bblayers.conf.sample b/meta-isar/conf/bblayers.conf.sample
index 80867e7..53a362b 100644
--- a/meta-isar/conf/bblayers.conf.sample
+++ b/meta-isar/conf/bblayers.conf.sample
@@ -8,6 +8,7 @@ BBFILES ?= ""
BBLAYERS ?= " \
##ISARROOT##/meta \
##ISARROOT##/meta-isar \
+ ##ISARROOT##/meta-isar-bin \
"
BBLAYERS_NON_REMOVABLE ?= " \
##ISARROOT##/meta \
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5bf9524..9ab9b19 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -10,6 +10,19 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
inherit ${IMAGE_TYPE}
+CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+
+do_cache_config() {
+ if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+ sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
+ ${DEBFILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+ fi
+}
+
+addtask cache_config before do_fetch
+
do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
# Install Debian packages, that were built from sources
--
2.1.4
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 2/4] meta-isar-bin: Generate cache repos
2017-09-25 9:59 [PATCH v3 0/4] Basic binary cache implementation Alexander Smirnov
2017-09-25 9:59 ` [PATCH v3 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov
@ 2017-09-25 9:59 ` Alexander Smirnov
2017-09-25 11:43 ` Henning Schild
2017-09-25 9:59 ` [PATCH v3 3/4] meta-isar-bin: Populate cache Alexander Smirnov
2017-09-25 10:00 ` [PATCH v3 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov
3 siblings, 1 reply; 10+ messages in thread
From: Alexander Smirnov @ 2017-09-25 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Generate repos for each configuration in multiconfig.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar-bin/conf/layer.conf | 3 +++
meta-isar/conf/local.conf.sample | 4 ++++
meta/classes/image.bbclass | 16 ++++++++++++++++
3 files changed, 23 insertions(+)
diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
index 3518184..5ef6e60 100644
--- a/meta-isar-bin/conf/layer.conf
+++ b/meta-isar-bin/conf/layer.conf
@@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
# Path to the configuration files templates used by `reprepro`
DEBFILESDIR ?= "${LAYERDIR}/files"
+
+# Path to the databases used by `reprepro`
+DEBDBDIR ?= "${LAYERDIR}/db"
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index a456b1b..660958f 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
#
# Default parallel jobs for bitbake:
BB_NUMBER_THREADS = "4"
+
+#
+# Number of attempts to try to get reprepro lock for access to apt cache
+REPREPRO_LOCK_ATTEMPTS = "16"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 9ab9b19..6f39c7a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -14,11 +14,27 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
do_cache_config[dirs] = "${CACHE_CONF_DIR}"
do_cache_config[stamp-extra-info] = "${DISTRO}"
+call_reprepro() {
+ for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
+ reprepro --waitforlock 5 $* \
+ || bbwarn "Failed to get reprepro lock, trying again..."
+ done
+}
+
do_cache_config() {
if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
${DEBFILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
fi
+
+ path_cache="${DEBCACHEDIR}/${DISTRO}"
+ path_databases="${DEBDBDIR}/${DISTRO}"
+
+ if [ ! -d "${path_databases}" ]; then
+ call_reprepro -b ${path_cache} \
+ --dbdir ${path_databases} \
+ export ${DEBDISTRONAME}
+ fi
}
addtask cache_config before do_fetch
--
2.1.4
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 3/4] meta-isar-bin: Populate cache
2017-09-25 9:59 [PATCH v3 0/4] Basic binary cache implementation Alexander Smirnov
2017-09-25 9:59 ` [PATCH v3 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov
2017-09-25 9:59 ` [PATCH v3 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov
@ 2017-09-25 9:59 ` Alexander Smirnov
2017-09-25 10:00 ` [PATCH v3 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov
3 siblings, 0 replies; 10+ messages in thread
From: Alexander Smirnov @ 2017-09-25 9:59 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Add newly built packages to apt cache.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta/classes/image.bbclass | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 6f39c7a..a079be7 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -44,6 +44,14 @@ do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
# Install Debian packages, that were built from sources
do_populate() {
if [ -n "${IMAGE_INSTALL}" ]; then
+ for p in ${IMAGE_INSTALL}; do
+ call_reprepro -b ${DEBCACHEDIR}/${DISTRO} \
+ --dbdir ${DEBDBDIR}/${DISTRO} \
+ -C main \
+ includedeb ${DEBDISTRONAME} \
+ ${DEPLOY_DIR_DEB}/${p}_*.deb
+ done
+
sudo mkdir -p ${IMAGE_ROOTFS}/deb
for p in ${IMAGE_INSTALL}; do
--
2.1.4
^ permalink raw reply [flat|nested] 10+ messages in thread
* [PATCH v3 4/4] meta-isar-bin: Install packages via multistrap
2017-09-25 9:59 [PATCH v3 0/4] Basic binary cache implementation Alexander Smirnov
` (2 preceding siblings ...)
2017-09-25 9:59 ` [PATCH v3 3/4] meta-isar-bin: Populate cache Alexander Smirnov
@ 2017-09-25 10:00 ` Alexander Smirnov
3 siblings, 0 replies; 10+ messages in thread
From: Alexander Smirnov @ 2017-09-25 10:00 UTC (permalink / raw)
To: isar-users; +Cc: Alexander Smirnov
Install custom packages to target image filesystem using multistrap.
Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
meta-isar/recipes-core/images/files/multistrap.conf.in | 9 ++++++++-
meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++-
meta/classes/ext4-img.bbclass | 2 +-
meta/classes/image.bbclass | 10 ----------
4 files changed, 13 insertions(+), 13 deletions(-)
diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..38d53cf 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
[General]
noauth=true
unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
aptsources=##DISTRO##
configscript=##CONFIG_SCRIPT##
setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
components=##DISTRO_COMPONENTS##
packages=##IMAGE_PREINSTALL##
omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_DEB##
+suite=##ISAR_DISTRO_SUITE##
+components=main
+omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 5cb8b1c..092abb2 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -43,6 +43,9 @@ do_rootfs() {
-e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
-e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
-e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+ -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+ -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' \
+ -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
${WORKDIR}/multistrap.conf
# Create root filesystem
@@ -54,4 +57,4 @@ do_rootfs() {
sudo rm ${IMAGE_ROOTFS}/configscript.sh
}
-addtask rootfs before do_populate
+addtask rootfs before do_build after do_populate
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
fi
}
-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index a079be7..de59ee2 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -51,16 +51,6 @@ do_populate() {
includedeb ${DEBDISTRONAME} \
${DEPLOY_DIR_DEB}/${p}_*.deb
done
-
- sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
- for p in ${IMAGE_INSTALL}; do
- sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
- done
-
- sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
- sudo rm -rf ${IMAGE_ROOTFS}/deb
fi
}
--
2.1.4
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/4] meta-isar-bin: Add reprepro configs
2017-09-25 9:59 ` [PATCH v3 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov
@ 2017-09-25 11:25 ` Henning Schild
2017-10-02 15:51 ` Alexander Smirnov
0 siblings, 1 reply; 10+ messages in thread
From: Henning Schild @ 2017-09-25 11:25 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Mon, 25 Sep 2017 12:59:57 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> Add template for reprepro configuration file and function that
> generates final config for specific deistro.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar-bin/conf/layer.conf | 11 +++++++++++
> meta-isar-bin/files/distributions.in | 3 +++
> meta-isar/conf/bblayers.conf.sample | 1 +
> meta/classes/image.bbclass | 13 +++++++++++++
> 4 files changed, 28 insertions(+)
> create mode 100644 meta-isar-bin/conf/layer.conf
> create mode 100644 meta-isar-bin/files/distributions.in
>
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf new file mode 100644
> index 0000000..3518184
> --- /dev/null
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -0,0 +1,11 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2017 ilbers GmbH
> +
> +# Codename of the repository created by the caching class
> +DEBDISTRONAME = "isar"
> +
> +# Path to the caching repository
> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> +
> +# Path to the configuration files templates used by `reprepro`
> +DEBFILESDIR ?= "${LAYERDIR}/files"
> diff --git a/meta-isar-bin/files/distributions.in
> b/meta-isar-bin/files/distributions.in new file mode 100644
> index 0000000..cd214c6
> --- /dev/null
> +++ b/meta-isar-bin/files/distributions.in
> @@ -0,0 +1,3 @@
> +Codename: {DISTRO_NAME}
> +Architectures: i386 armhf amd64 source
> +Components: main
> diff --git a/meta-isar/conf/bblayers.conf.sample
> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> --- a/meta-isar/conf/bblayers.conf.sample
> +++ b/meta-isar/conf/bblayers.conf.sample
> @@ -8,6 +8,7 @@ BBFILES ?= ""
> BBLAYERS ?= " \
> ##ISARROOT##/meta \
> ##ISARROOT##/meta-isar \
> + ##ISARROOT##/meta-isar-bin \
> "
> BBLAYERS_NON_REMOVABLE ?= " \
> ##ISARROOT##/meta \
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 5bf9524..9ab9b19 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -10,6 +10,19 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
>
> inherit ${IMAGE_TYPE}
>
> +CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
> +do_cache_config[dirs] = "${CACHE_CONF_DIR}"
> +do_cache_config[stamp-extra-info] = "${DISTRO}"
> +
> +do_cache_config() {
> + if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
> + sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
> + ${DEBFILESDIR}/distributions.in >
> ${CACHE_CONF_DIR}/distributions
> + fi
> +}
> +
> +addtask cache_config before do_fetch
Looking at the patch from Claudius, this one should probably use the
fetcher for the .in-file as well. The step would naturally go after
do_unpack.
Henning
> do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>
> # Install Debian packages, that were built from sources
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos
2017-09-25 9:59 ` [PATCH v3 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov
@ 2017-09-25 11:43 ` Henning Schild
2017-09-25 12:31 ` Alexander Smirnov
0 siblings, 1 reply; 10+ messages in thread
From: Henning Schild @ 2017-09-25 11:43 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Mon, 25 Sep 2017 12:59:58 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> Generate repos for each configuration in multiconfig.
>
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
> meta-isar-bin/conf/layer.conf | 3 +++
> meta-isar/conf/local.conf.sample | 4 ++++
> meta/classes/image.bbclass | 16 ++++++++++++++++
> 3 files changed, 23 insertions(+)
>
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
> --- a/meta-isar-bin/conf/layer.conf
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>
> # Path to the configuration files templates used by `reprepro`
> DEBFILESDIR ?= "${LAYERDIR}/files"
> +
> +# Path to the databases used by `reprepro`
> +DEBDBDIR ?= "${LAYERDIR}/db"
> diff --git a/meta-isar/conf/local.conf.sample
> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
> #
> # Default parallel jobs for bitbake:
> BB_NUMBER_THREADS = "4"
> +
> +#
> +# Number of attempts to try to get reprepro lock for access to apt
> cache +REPREPRO_LOCK_ATTEMPTS = "16"
>
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 9ab9b19..6f39c7a 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -14,11 +14,27 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
> do_cache_config[dirs] = "${CACHE_CONF_DIR}"
> do_cache_config[stamp-extra-info] = "${DISTRO}"
>
> +call_reprepro() {
> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> + reprepro --waitforlock 5 $* \
> + || bbwarn "Failed to get reprepro lock, trying again..."
Now we have two loops and the number from above need to be bultiplied
with the inner number .. 5*16 times 10s. I still do not see how that
solves the problem. But i can imagine those two loops really slowing
down builds that fail. Looks like they potentially add 800s build-time.
And every error of reprepro is turned into "Failed to get lock" ... As
far as i looked at reprepro it should return EEXIST in the
contention case.
Henning
> + done
> +}
> +
> do_cache_config() {
> if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
> ${DEBFILESDIR}/distributions.in >
> ${CACHE_CONF_DIR}/distributions fi
> +
> + path_cache="${DEBCACHEDIR}/${DISTRO}"
> + path_databases="${DEBDBDIR}/${DISTRO}"
> +
> + if [ ! -d "${path_databases}" ]; then
> + call_reprepro -b ${path_cache} \
> + --dbdir ${path_databases} \
> + export ${DEBDISTRONAME}
> + fi
> }
>
> addtask cache_config before do_fetch
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos
2017-09-25 11:43 ` Henning Schild
@ 2017-09-25 12:31 ` Alexander Smirnov
2017-09-25 13:52 ` Henning Schild
0 siblings, 1 reply; 10+ messages in thread
From: Alexander Smirnov @ 2017-09-25 12:31 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 09/25/2017 02:43 PM, Henning Schild wrote:
> Am Mon, 25 Sep 2017 12:59:58 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> Generate repos for each configuration in multiconfig.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta-isar-bin/conf/layer.conf | 3 +++
>> meta-isar/conf/local.conf.sample | 4 ++++
>> meta/classes/image.bbclass | 16 ++++++++++++++++
>> 3 files changed, 23 insertions(+)
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
>> --- a/meta-isar-bin/conf/layer.conf
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>
>> # Path to the configuration files templates used by `reprepro`
>> DEBFILESDIR ?= "${LAYERDIR}/files"
>> +
>> +# Path to the databases used by `reprepro`
>> +DEBDBDIR ?= "${LAYERDIR}/db"
>> diff --git a/meta-isar/conf/local.conf.sample
>> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
>> #
>> # Default parallel jobs for bitbake:
>> BB_NUMBER_THREADS = "4"
>> +
>> +#
>> +# Number of attempts to try to get reprepro lock for access to apt
>> cache +REPREPRO_LOCK_ATTEMPTS = "16"
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 9ab9b19..6f39c7a 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -14,11 +14,27 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
>> do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>> do_cache_config[stamp-extra-info] = "${DISTRO}"
>>
>> +call_reprepro() {
>> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
>> + reprepro --waitforlock 5 $* \
>> + || bbwarn "Failed to get reprepro lock, trying again..."
>
> Now we have two loops and the number from above need to be bultiplied
> with the inner number .. 5*16 times 10s. I still do not see how that
> solves the problem. But i can imagine those two loops really slowing
> down builds that fail. Looks like they potentially add 800s build-time.
> And every error of reprepro is turned into "Failed to get lock" ... As
> far as i looked at reprepro it should return EEXIST in the
> contention case.
>
This is the compromise to avoid races in reprepro. Some thoughts:
1. Infinite loop is not an option, because I expect that broken build
will eventually exit.
2. I want to have some notifications if lock acquisition fails for a
long time.
3. Possible number of races depends on amount of packages to be
processed. So this variable should be tuned anyway.
Until reprepro can't query processing from different threads and bitbake
doesn't provide lock primitives, I don't see how this could be solved
permanently at the moment.
I've checked reprepro return code:
$ reprepro -b meta-isar-bin/apt/debian-stretch/ --dbdir
./meta-isar-bin/db/debian-stretch/ -C main ls hello || echo $?
...
239
So looks like the code is really EEXIST. In this case I'd like to add
check for this exitcode and keep the loop as it is. Any other suggestions?
Alex
> Henning
>
>> + done
>> +}
>> +
>> do_cache_config() {
>> if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
>> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>> ${DEBFILESDIR}/distributions.in >
>> ${CACHE_CONF_DIR}/distributions fi
>> +
>> + path_cache="${DEBCACHEDIR}/${DISTRO}"
>> + path_databases="${DEBDBDIR}/${DISTRO}"
>> +
>> + if [ ! -d "${path_databases}" ]; then
>> + call_reprepro -b ${path_cache} \
>> + --dbdir ${path_databases} \
>> + export ${DEBDISTRONAME}
>> + fi
>> }
>>
>> addtask cache_config before do_fetch
>
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 2/4] meta-isar-bin: Generate cache repos
2017-09-25 12:31 ` Alexander Smirnov
@ 2017-09-25 13:52 ` Henning Schild
0 siblings, 0 replies; 10+ messages in thread
From: Henning Schild @ 2017-09-25 13:52 UTC (permalink / raw)
To: Alexander Smirnov; +Cc: isar-users
Am Mon, 25 Sep 2017 15:31:13 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> On 09/25/2017 02:43 PM, Henning Schild wrote:
> > Am Mon, 25 Sep 2017 12:59:58 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >
> >> Generate repos for each configuration in multiconfig.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >> meta-isar-bin/conf/layer.conf | 3 +++
> >> meta-isar/conf/local.conf.sample | 4 ++++
> >> meta/classes/image.bbclass | 16 ++++++++++++++++
> >> 3 files changed, 23 insertions(+)
> >>
> >> diff --git a/meta-isar-bin/conf/layer.conf
> >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
> >> --- a/meta-isar-bin/conf/layer.conf
> >> +++ b/meta-isar-bin/conf/layer.conf
> >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
> >>
> >> # Path to the configuration files templates used by `reprepro`
> >> DEBFILESDIR ?= "${LAYERDIR}/files"
> >> +
> >> +# Path to the databases used by `reprepro`
> >> +DEBDBDIR ?= "${LAYERDIR}/db"
> >> diff --git a/meta-isar/conf/local.conf.sample
> >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
> >> --- a/meta-isar/conf/local.conf.sample
> >> +++ b/meta-isar/conf/local.conf.sample
> >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
> >> #
> >> # Default parallel jobs for bitbake:
> >> BB_NUMBER_THREADS = "4"
> >> +
> >> +#
> >> +# Number of attempts to try to get reprepro lock for access to apt
> >> cache +REPREPRO_LOCK_ATTEMPTS = "16"
> >>
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 9ab9b19..6f39c7a 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -14,11 +14,27 @@ CACHE_CONF_DIR =
> >> "${DEBCACHEDIR}/${DISTRO}/conf" do_cache_config[dirs] =
> >> "${CACHE_CONF_DIR}" do_cache_config[stamp-extra-info] = "${DISTRO}"
> >>
> >> +call_reprepro() {
> >> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> >> + reprepro --waitforlock 5 $* \
> >> + || bbwarn "Failed to get reprepro lock, trying again..."
> >
> > Now we have two loops and the number from above need to be
> > bultiplied with the inner number .. 5*16 times 10s. I still do not
> > see how that solves the problem. But i can imagine those two loops
> > really slowing down builds that fail. Looks like they potentially
> > add 800s build-time. And every error of reprepro is turned into
> > "Failed to get lock" ... As far as i looked at reprepro it should
> > return EEXIST in the contention case.
> >
>
> This is the compromise to avoid races in reprepro. Some thoughts:
>
> 1. Infinite loop is not an option, because I expect that broken build
> will eventually exit.
>
> 2. I want to have some notifications if lock acquisition fails for a
> long time.
>
> 3. Possible number of races depends on amount of packages to be
> processed. So this variable should be tuned anyway.
>
> Until reprepro can't query processing from different threads and
> bitbake doesn't provide lock primitives, I don't see how this could
> be solved permanently at the moment.
>
> I've checked reprepro return code:
>
> $ reprepro -b meta-isar-bin/apt/debian-stretch/ --dbdir
> ./meta-isar-bin/db/debian-stretch/ -C main ls hello || echo $?
> ...
> 239
EEXIST should be 17 and this looks like a uint8 -EEXIST, best use
python for getting the number
import errno; print(errno.EEXIST)
> So looks like the code is really EEXIST. In this case I'd like to add
> check for this exitcode and keep the loop as it is. Any other
> suggestions?
Yes, the inner loop should use a "--waitforlock 1" instead of 5.
Henning
> Alex
>
> > Henning
> >
> >> + done
> >> +}
> >> +
> >> do_cache_config() {
> >> if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
> >> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
> >> ${DEBFILESDIR}/distributions.in >
> >> ${CACHE_CONF_DIR}/distributions fi
> >> +
> >> + path_cache="${DEBCACHEDIR}/${DISTRO}"
> >> + path_databases="${DEBDBDIR}/${DISTRO}"
> >> +
> >> + if [ ! -d "${path_databases}" ]; then
> >> + call_reprepro -b ${path_cache} \
> >> + --dbdir ${path_databases} \
> >> + export ${DEBDISTRONAME}
> >> + fi
> >> }
> >>
> >> addtask cache_config before do_fetch
> >
>
^ permalink raw reply [flat|nested] 10+ messages in thread
* Re: [PATCH v3 1/4] meta-isar-bin: Add reprepro configs
2017-09-25 11:25 ` Henning Schild
@ 2017-10-02 15:51 ` Alexander Smirnov
0 siblings, 0 replies; 10+ messages in thread
From: Alexander Smirnov @ 2017-10-02 15:51 UTC (permalink / raw)
To: Henning Schild; +Cc: isar-users
On 09/25/2017 02:25 PM, Henning Schild wrote:
> Am Mon, 25 Sep 2017 12:59:57 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>
>> Add template for reprepro configuration file and function that
>> generates final config for specific deistro.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>> meta-isar-bin/conf/layer.conf | 11 +++++++++++
>> meta-isar-bin/files/distributions.in | 3 +++
>> meta-isar/conf/bblayers.conf.sample | 1 +
>> meta/classes/image.bbclass | 13 +++++++++++++
>> 4 files changed, 28 insertions(+)
>> create mode 100644 meta-isar-bin/conf/layer.conf
>> create mode 100644 meta-isar-bin/files/distributions.in
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf new file mode 100644
>> index 0000000..3518184
>> --- /dev/null
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -0,0 +1,11 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2017 ilbers GmbH
>> +
>> +# Codename of the repository created by the caching class
>> +DEBDISTRONAME = "isar"
>> +
>> +# Path to the caching repository
>> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
>> +
>> +# Path to the configuration files templates used by `reprepro`
>> +DEBFILESDIR ?= "${LAYERDIR}/files"
>> diff --git a/meta-isar-bin/files/distributions.in
>> b/meta-isar-bin/files/distributions.in new file mode 100644
>> index 0000000..cd214c6
>> --- /dev/null
>> +++ b/meta-isar-bin/files/distributions.in
>> @@ -0,0 +1,3 @@
>> +Codename: {DISTRO_NAME}
>> +Architectures: i386 armhf amd64 source
>> +Components: main
>> diff --git a/meta-isar/conf/bblayers.conf.sample
>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
>> --- a/meta-isar/conf/bblayers.conf.sample
>> +++ b/meta-isar/conf/bblayers.conf.sample
>> @@ -8,6 +8,7 @@ BBFILES ?= ""
>> BBLAYERS ?= " \
>> ##ISARROOT##/meta \
>> ##ISARROOT##/meta-isar \
>> + ##ISARROOT##/meta-isar-bin \
>> "
>> BBLAYERS_NON_REMOVABLE ?= " \
>> ##ISARROOT##/meta \
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 5bf9524..9ab9b19 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -10,6 +10,19 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs"
>>
>> inherit ${IMAGE_TYPE}
>>
>> +CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
>> +do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>> +do_cache_config[stamp-extra-info] = "${DISTRO}"
>> +
>> +do_cache_config() {
>> + if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
>> + sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>> + ${DEBFILESDIR}/distributions.in >
>> ${CACHE_CONF_DIR}/distributions
>> + fi
>> +}
>> +
>> +addtask cache_config before do_fetch
>
> Looking at the patch from Claudius, this one should probably use the
> fetcher for the .in-file as well. The step would naturally go after
> do_unpack.
>
I've thought about this and tried to implement, but seems it's not the
case here:
- Default fetch task for each image will copy template to each image
workspace, while it's needed only if there is no already generated config.
- This config should be generated once per-distro.
So I think this should stay like it is for now.
Alex
> Henning
>
>> do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>
>> # Install Debian packages, that were built from sources
>
--
With best regards,
Alexander Smirnov
ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov
^ permalink raw reply [flat|nested] 10+ messages in thread
end of thread, other threads:[~2017-10-02 15:51 UTC | newest]
Thread overview: 10+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-09-25 9:59 [PATCH v3 0/4] Basic binary cache implementation Alexander Smirnov
2017-09-25 9:59 ` [PATCH v3 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov
2017-09-25 11:25 ` Henning Schild
2017-10-02 15:51 ` Alexander Smirnov
2017-09-25 9:59 ` [PATCH v3 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov
2017-09-25 11:43 ` Henning Schild
2017-09-25 12:31 ` Alexander Smirnov
2017-09-25 13:52 ` Henning Schild
2017-09-25 9:59 ` [PATCH v3 3/4] meta-isar-bin: Populate cache Alexander Smirnov
2017-09-25 10:00 ` [PATCH v3 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox