[PATCH v2 0/4] Basic binary cache implementation

[PATCH v2 0/4] Basic binary cache implementation

[Alexander Smirnov]

Hello everybody,

this series introduces basic binary caching for Isar.

There is a new layer: meta-isar-bin which is intended to be the binary
cache. All the packages that are built in Isar are stored in this
cache using reprepro.

Having the separate layer makes possible to manage this cache separately
from the Isar build tree. So once you have built your packages, you could
re-use this cache for further builds.

Also this cache repo is now used to generate image filesystem via
multistrap.

Changes since v1:
 - Total refactoring
 - Split huge patches into small pieces
 - Avoid races

With best regards,
Alex

8<--

Alexander Smirnov (4):
  meta-isar-bin: Add reprepro configs
  meta-isar-bin: Generate cache repos
  meta-isar-bin: Populate cache
  meta-isar-bin: Install packages via multistrap

 meta-isar-bin/conf/layer.conf                      | 14 ++++++
 meta-isar-bin/files/distributions.in               |  3 ++
 meta-isar/conf/bblayers.conf.sample                |  1 +
 .../recipes-core/images/files/multistrap.conf.in   |  9 +++-
 meta-isar/recipes-core/images/isar-image-base.bb   |  5 ++-
 meta/classes/ext4-img.bbclass                      |  2 +-
 meta/classes/image.bbclass                         | 52 +++++++++++++++++++---
 7 files changed, 76 insertions(+), 10 deletions(-)
 create mode 100644 meta-isar-bin/conf/layer.conf
 create mode 100644 meta-isar-bin/files/distributions.in

-- 
2.1.4

[PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Alexander Smirnov]

Add template for reprepro configuration file and function that
generates final config for specific deistro.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar-bin/conf/layer.conf        | 11 +++++++++++
 meta-isar-bin/files/distributions.in |  3 +++
 meta-isar/conf/bblayers.conf.sample  |  1 +
 meta/classes/image.bbclass           | 25 +++++++++++++++++++++++++
 4 files changed, 40 insertions(+)
 create mode 100644 meta-isar-bin/conf/layer.conf
 create mode 100644 meta-isar-bin/files/distributions.in

diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
new file mode 100644
index 0000000..3518184
--- /dev/null
+++ b/meta-isar-bin/conf/layer.conf
@@ -0,0 +1,11 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017 ilbers GmbH
+
+# Codename of the repository created by the caching class
+DEBDISTRONAME = "isar"
+
+# Path to the caching repository
+DEBCACHEDIR ?= "${LAYERDIR}/apt"
+
+# Path to the configuration files templates used by `reprepro`
+DEBFILESDIR ?= "${LAYERDIR}/files"
diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta-isar-bin/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta-isar/conf/bblayers.conf.sample b/meta-isar/conf/bblayers.conf.sample
index 80867e7..53a362b 100644
--- a/meta-isar/conf/bblayers.conf.sample
+++ b/meta-isar/conf/bblayers.conf.sample
@@ -8,6 +8,7 @@ BBFILES ?= ""
 BBLAYERS ?= " \
   ##ISARROOT##/meta \
   ##ISARROOT##/meta-isar \
+  ##ISARROOT##/meta-isar-bin \
   "
 BBLAYERS_NON_REMOVABLE ?= " \
   ##ISARROOT##/meta \
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5bf9524..30f241a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
 
 inherit ${IMAGE_TYPE}
 
+# Generate apt cache configuration file
+python __anonymous () {
+    DISTRO = d.getVar("DISTRO", True)
+    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
+    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
+
+    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
+    cache_conf_file = os.path.join(cache_conf_path, "distributions")
+    cache_conf_file_in = os.path.join(DEBFILESDIR, "distributions.in")
+
+    import subprocess
+
+    if not os.path.exists(cache_conf_path):
+        os.makedirs(cache_conf_path)
+
+    if not os.path.exists(cache_conf_file):
+        f = open(cache_conf_file, "w")
+        subprocess.check_call([
+            "sed",
+            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME", True) + "#g",
+            cache_conf_file_in,
+            ], stdout=f)
+        f.close()
+}
+
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 
 # Install Debian packages, that were built from sources
-- 
2.1.4

[PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov]

Generate repos for each configuration in multiconfig.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar-bin/conf/layer.conf |  3 +++
 meta/classes/image.bbclass    | 14 ++++++++++++++
 2 files changed, 17 insertions(+)

diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
index 3518184..5ef6e60 100644
--- a/meta-isar-bin/conf/layer.conf
+++ b/meta-isar-bin/conf/layer.conf
@@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
 
 # Path to the configuration files templates used by `reprepro`
 DEBFILESDIR ?= "${LAYERDIR}/files"
+
+# Path to the databases used by `reprepro`
+DEBDBDIR ?= "${LAYERDIR}/db"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 30f241a..9db7371 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -35,6 +35,20 @@ python __anonymous () {
         f.close()
 }
 
+do_create_cache_db[stamp-extra-info] = "${DISTRO}"
+
+do_create_cache_db() {
+    path_cache="${DEBCACHEDIR}/${DISTRO}"
+    path_databases="${DEBDBDIR}/${DISTRO}"
+
+    if [ ! -d "${path_databases}" ]; then
+        reprepro --waitforlock 3 -b ${path_cache} --dbdir ${path_databases} \
+                 export ${DEBDISTRONAME}
+    fi
+}
+
+addtask create_cache_db before do_fetch
+
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 
 # Install Debian packages, that were built from sources
-- 
2.1.4

[PATCH v2 3/4] meta-isar-bin: Populate cache

[Alexander Smirnov]

Add newly built packages to apt cache.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/image.bbclass | 9 +++++++++
 1 file changed, 9 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 9db7371..d59959b 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -54,6 +54,15 @@ do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 # Install Debian packages, that were built from sources
 do_populate() {
     if [ -n "${IMAGE_INSTALL}" ]; then
+        for p in ${IMAGE_INSTALL}; do
+            reprepro --waitforlock 3 \
+                     -b ${DEBCACHEDIR}/${DISTRO} \
+                     --dbdir ${DEBDBDIR}/${DISTRO} \
+                     -C main \
+                     includedeb ${DEBDISTRONAME} \
+                     ${DEPLOY_DIR_DEB}/${p}_*.deb
+        done
+
         sudo mkdir -p ${IMAGE_ROOTFS}/deb
 
         for p in ${IMAGE_INSTALL}; do
-- 
2.1.4

[PATCH v2 4/4] meta-isar-bin: Install packages via multistrap

[Alexander Smirnov]

Install custom packages to target image filesystem using multistrap.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
 meta-isar/recipes-core/images/isar-image-base.bb       |  5 ++++-
 meta/classes/ext4-img.bbclass                          |  2 +-
 meta/classes/image.bbclass                             | 10 ----------
 4 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..38d53cf 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
 [General]
 noauth=true
 unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
 aptsources=##DISTRO##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
 components=##DISTRO_COMPONENTS##
 packages=##IMAGE_PREINSTALL##
 omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_DEB##
+suite=##ISAR_DISTRO_SUITE##
+components=main
+omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index 5cb8b1c..092abb2 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -43,6 +43,9 @@ do_rootfs() {
            -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \
            -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
            -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+           -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+           -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' \
+           -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
               ${WORKDIR}/multistrap.conf
 
     # Create root filesystem
@@ -54,4 +57,4 @@ do_rootfs() {
     sudo rm ${IMAGE_ROOTFS}/configscript.sh
 }
 
-addtask rootfs before do_populate
+addtask rootfs before do_build after do_populate
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
     fi
 }
 
-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index d59959b..410f30f 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -62,16 +62,6 @@ do_populate() {
                      includedeb ${DEBDISTRONAME} \
                      ${DEPLOY_DIR_DEB}/${p}_*.deb
         done
-
-        sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
-        for p in ${IMAGE_INSTALL}; do
-            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
-        done
-
-        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
-        sudo rm -rf ${IMAGE_ROOTFS}/deb
     fi
 }
 
-- 
2.1.4

Re: [PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Henning Schild]

Am Tue, 19 Sep 2017 15:20:49 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add template for reprepro configuration file and function that
> generates final config for specific deistro.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar-bin/conf/layer.conf        | 11 +++++++++++
>  meta-isar-bin/files/distributions.in |  3 +++
>  meta-isar/conf/bblayers.conf.sample  |  1 +
>  meta/classes/image.bbclass           | 25 +++++++++++++++++++++++++
>  4 files changed, 40 insertions(+)
>  create mode 100644 meta-isar-bin/conf/layer.conf
>  create mode 100644 meta-isar-bin/files/distributions.in
> 
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf new file mode 100644
> index 0000000..3518184
> --- /dev/null
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -0,0 +1,11 @@
> +# This software is a part of ISAR.
> +# Copyright (C) 2017 ilbers GmbH
> +
> +# Codename of the repository created by the caching class
> +DEBDISTRONAME = "isar"
> +
> +# Path to the caching repository
> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> +
> +# Path to the configuration files templates used by `reprepro`
> +DEBFILESDIR ?= "${LAYERDIR}/files"
> diff --git a/meta-isar-bin/files/distributions.in
> b/meta-isar-bin/files/distributions.in new file mode 100644
> index 0000000..cd214c6
> --- /dev/null
> +++ b/meta-isar-bin/files/distributions.in
> @@ -0,0 +1,3 @@
> +Codename: {DISTRO_NAME}
> +Architectures: i386 armhf amd64 source
> +Components: main
> diff --git a/meta-isar/conf/bblayers.conf.sample
> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> --- a/meta-isar/conf/bblayers.conf.sample
> +++ b/meta-isar/conf/bblayers.conf.sample
> @@ -8,6 +8,7 @@ BBFILES ?= ""
>  BBLAYERS ?= " \
>    ##ISARROOT##/meta \
>    ##ISARROOT##/meta-isar \
> +  ##ISARROOT##/meta-isar-bin \
>    "
>  BBLAYERS_NON_REMOVABLE ?= " \
>    ##ISARROOT##/meta \
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 5bf9524..30f241a 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
>  
>  inherit ${IMAGE_TYPE}
>  
> +# Generate apt cache configuration file
> +python __anonymous () {
> +    DISTRO = d.getVar("DISTRO", True)
> +    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
> +    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
> +
> +    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
> +    cache_conf_file = os.path.join(cache_conf_path, "distributions")
> +    cache_conf_file_in = os.path.join(DEBFILESDIR,
> "distributions.in") +
> +    import subprocess
> +
> +    if not os.path.exists(cache_conf_path):
> +        os.makedirs(cache_conf_path)
> +
> +    if not os.path.exists(cache_conf_file):
> +        f = open(cache_conf_file, "w")
> +        subprocess.check_call([
> +            "sed",
> +            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME",
> True) + "#g",
> +            cache_conf_file_in,
> +            ], stdout=f)
> +        f.close()
> +}

Why is this not just a regular task early in the image chain?

Henning

>  do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>  
>  # Install Debian packages, that were built from sources

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Henning Schild]

Am Tue, 19 Sep 2017 15:20:50 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Generate repos for each configuration in multiconfig.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar-bin/conf/layer.conf |  3 +++
>  meta/classes/image.bbclass    | 14 ++++++++++++++
>  2 files changed, 17 insertions(+)
> 
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
> --- a/meta-isar-bin/conf/layer.conf
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>  
>  # Path to the configuration files templates used by `reprepro`
>  DEBFILESDIR ?= "${LAYERDIR}/files"
> +
> +# Path to the databases used by `reprepro`
> +DEBDBDIR ?= "${LAYERDIR}/db"
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 30f241a..9db7371 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -35,6 +35,20 @@ python __anonymous () {
>          f.close()
>  }
>  
> +do_create_cache_db[stamp-extra-info] = "${DISTRO}"
> +
> +do_create_cache_db() {
> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
> +    path_databases="${DEBDBDIR}/${DISTRO}"
> +
> +    if [ ! -d "${path_databases}" ]; then
> +        reprepro --waitforlock 3 -b ${path_cache} --dbdir
> ${path_databases} \
> +                 export ${DEBDISTRONAME}
> +    fi
> +}
> +
> +addtask create_cache_db before do_fetch

This adds a new tool dependency, maybe update docs with the patch or
keep in mind for later.

Why "waitforlock 3" is there any chance at all that multiple images run
into that at the same time? And if so how do we know 3 is enough? I just
tested that a second "reprepro export" will still return 0, so it seems
safe to do that multiple times, not sure what would happen if some
packages where already added and you "export" again.

Henning

>  do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>  
>  # Install Debian packages, that were built from sources

Re: [PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Alexander Smirnov]

On 09/20/2017 10:58 AM, Henning Schild wrote:
> Am Tue, 19 Sep 2017 15:20:49 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Add template for reprepro configuration file and function that
>> generates final config for specific deistro.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar-bin/conf/layer.conf        | 11 +++++++++++
>>   meta-isar-bin/files/distributions.in |  3 +++
>>   meta-isar/conf/bblayers.conf.sample  |  1 +
>>   meta/classes/image.bbclass           | 25 +++++++++++++++++++++++++
>>   4 files changed, 40 insertions(+)
>>   create mode 100644 meta-isar-bin/conf/layer.conf
>>   create mode 100644 meta-isar-bin/files/distributions.in
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf new file mode 100644
>> index 0000000..3518184
>> --- /dev/null
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -0,0 +1,11 @@
>> +# This software is a part of ISAR.
>> +# Copyright (C) 2017 ilbers GmbH
>> +
>> +# Codename of the repository created by the caching class
>> +DEBDISTRONAME = "isar"
>> +
>> +# Path to the caching repository
>> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
>> +
>> +# Path to the configuration files templates used by `reprepro`
>> +DEBFILESDIR ?= "${LAYERDIR}/files"
>> diff --git a/meta-isar-bin/files/distributions.in
>> b/meta-isar-bin/files/distributions.in new file mode 100644
>> index 0000000..cd214c6
>> --- /dev/null
>> +++ b/meta-isar-bin/files/distributions.in
>> @@ -0,0 +1,3 @@
>> +Codename: {DISTRO_NAME}
>> +Architectures: i386 armhf amd64 source
>> +Components: main
>> diff --git a/meta-isar/conf/bblayers.conf.sample
>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
>> --- a/meta-isar/conf/bblayers.conf.sample
>> +++ b/meta-isar/conf/bblayers.conf.sample
>> @@ -8,6 +8,7 @@ BBFILES ?= ""
>>   BBLAYERS ?= " \
>>     ##ISARROOT##/meta \
>>     ##ISARROOT##/meta-isar \
>> +  ##ISARROOT##/meta-isar-bin \
>>     "
>>   BBLAYERS_NON_REMOVABLE ?= " \
>>     ##ISARROOT##/meta \
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 5bf9524..30f241a 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
>>   
>>   inherit ${IMAGE_TYPE}
>>   
>> +# Generate apt cache configuration file
>> +python __anonymous () {
>> +    DISTRO = d.getVar("DISTRO", True)
>> +    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
>> +    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
>> +
>> +    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
>> +    cache_conf_file = os.path.join(cache_conf_path, "distributions")
>> +    cache_conf_file_in = os.path.join(DEBFILESDIR,
>> "distributions.in") +
>> +    import subprocess
>> +
>> +    if not os.path.exists(cache_conf_path):
>> +        os.makedirs(cache_conf_path)
>> +
>> +    if not os.path.exists(cache_conf_file):
>> +        f = open(cache_conf_file, "w")
>> +        subprocess.check_call([
>> +            "sed",
>> +            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME",
>> True) + "#g",
>> +            cache_conf_file_in,
>> +            ], stdout=f)
>> +        f.close()
>> +}
> 
> Why is this not just a regular task early in the image chain?
> 

The idea is to avoid races with multiconfig. If I add this as a regular 
task, it could be run in parallel by different multiconfigs, so there 
could occur races in `mkdir` and `sed`. The same is true when you try to 
build several images at the same time:

   $ bitbake image1 image2

IIRC bitbake anonymous tasks are executed during parsing stage, so they 
are serialized.

Alex

> Henning
> 
>>   do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>   
>>   # Install Debian packages, that were built from sources
>

Re: [PATCH v2 3/4] meta-isar-bin: Populate cache

[Henning Schild]

Am Tue, 19 Sep 2017 15:20:51 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Add newly built packages to apt cache.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/image.bbclass | 9 +++++++++
>  1 file changed, 9 insertions(+)
> 
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 9db7371..d59959b 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -54,6 +54,15 @@ do_populate[stamp-extra-info] =
> "${DISTRO}-${MACHINE}" # Install Debian packages, that were built
> from sources do_populate() {
>      if [ -n "${IMAGE_INSTALL}" ]; then
> +        for p in ${IMAGE_INSTALL}; do
> +            reprepro --waitforlock 3 \
> +                     -b ${DEBCACHEDIR}/${DISTRO} \
> +                     --dbdir ${DEBDBDIR}/${DISTRO} \
> +                     -C main \
> +                     includedeb ${DEBDISTRONAME} \
> +                     ${DEPLOY_DIR_DEB}/${p}_*.deb
> +        done

Why not just do that in do_deploy_deb? Maybe DEPLOY_DIR_DEB could go
away because we deploy directly into the repo.

I found that for kernels built with "make deb-pkg" the "${p}_*.deb"
pattern does not match. At least the way that i named the recipe.
"do_deploy_deb" already works on WORKDIR/*.deb which naturally covers
all packages coming out of a recipe, even if they had names different
from the recipe. Imagine a recipe that downloads multiple .debs from
somewhere and they all have very different names.

Again the question about the waitforlock, 3 seems only slightly better
than 1. I think we need infinite or a serialization mechanism and 1.

Henning

>          sudo mkdir -p ${IMAGE_ROOTFS}/deb
>  
>          for p in ${IMAGE_INSTALL}; do

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov]

On 09/20/2017 11:11 AM, Henning Schild wrote:
> Am Tue, 19 Sep 2017 15:20:50 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Generate repos for each configuration in multiconfig.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar-bin/conf/layer.conf |  3 +++
>>   meta/classes/image.bbclass    | 14 ++++++++++++++
>>   2 files changed, 17 insertions(+)
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
>> --- a/meta-isar-bin/conf/layer.conf
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>   
>>   # Path to the configuration files templates used by `reprepro`
>>   DEBFILESDIR ?= "${LAYERDIR}/files"
>> +
>> +# Path to the databases used by `reprepro`
>> +DEBDBDIR ?= "${LAYERDIR}/db"
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 30f241a..9db7371 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -35,6 +35,20 @@ python __anonymous () {
>>           f.close()
>>   }
>>   
>> +do_create_cache_db[stamp-extra-info] = "${DISTRO}"
>> +
>> +do_create_cache_db() {
>> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
>> +    path_databases="${DEBDBDIR}/${DISTRO}"
>> +
>> +    if [ ! -d "${path_databases}" ]; then
>> +        reprepro --waitforlock 3 -b ${path_cache} --dbdir
>> ${path_databases} \
>> +                 export ${DEBDISTRONAME}
>> +    fi
>> +}
>> +
>> +addtask create_cache_db before do_fetch
> 
> This adds a new tool dependency, maybe update docs with the patch or
> keep in mind for later.

For sure, I'll document everything before merge, when we agree on 
general architecture.

> 
> Why "waitforlock 3" is there any chance at all that multiple images run
> into that at the same time? And if so how do we know 3 is enough? I just
> tested that a second "reprepro export" will still return 0, so it seems
> safe to do that multiple times, not sure what would happen if some
> packages where already added and you "export" again.

Yes, this task could be run multiple times, for example by the following 
line:

bitbake \
   multiconfig:qemuarm-jessie:isar-image-base \
   multiconfig:qemui386-jessie:isar-image-base \
   multiconfig:qemuamd64-jessie:isar-image-base

I'll start jessie build for 3 architectures:
  - arm
  - i386
  - amd64

After build, there will be the following in the cache (example for hello):

$ ls -1 meta-isar-bin/apt/debian-jessie/pool/main/h/hello/
hello_0.1+g7f35942-1_amd64.deb
hello_0.1+g7f35942-1_armhf.deb
hello_0.1+g7f35942-1_i386.deb

So using waitforlock is an attempt to protect reprepro from crashing. In 
general this will never happen for single-distro single-arch builds.

Alex

> 
> Henning
> 
>>   do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>   
>>   # Install Debian packages, that were built from sources
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v2 4/4] meta-isar-bin: Install packages via multistrap

[Henning Schild]

I like that one, it solves the debian depends problem. That one
probably deserves mentioning in the documentation, i also see it as the
"the only way for _anything_ into the rootfs, is a .deb and its hooks"
We still have the configscript that messes around, that can be
addressed with more patches on top.

Henning

Am Tue, 19 Sep 2017 15:20:52 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Install custom packages to target image filesystem using multistrap.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
>  meta-isar/recipes-core/images/isar-image-base.bb       |  5 ++++-
>  meta/classes/ext4-img.bbclass                          |  2 +-
>  meta/classes/image.bbclass                             | 10
> ---------- 4 files changed, 13 insertions(+), 13 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
> b/meta-isar/recipes-core/images/files/multistrap.conf.in index
> 2d08c36..38d53cf 100644 ---
> a/meta-isar/recipes-core/images/files/multistrap.conf.in +++
> b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7
> @@ [General]
>  noauth=true
>  unpack=true
> -bootstrap=##DISTRO##
> +bootstrap=##DISTRO## Isar
>  aptsources=##DISTRO##
>  configscript=##CONFIG_SCRIPT##
>  setupscript=##SETUP_SCRIPT##
> @@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
>  components=##DISTRO_COMPONENTS##
>  packages=##IMAGE_PREINSTALL##
>  omitdebsrc=true
> +
> +[Isar]
> +packages=##IMAGE_INSTALL##
> +source=##DEPLOY_DIR_DEB##
> +suite=##ISAR_DISTRO_SUITE##
> +components=main
> +omitdebsrc=true
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> 5cb8b1c..092abb2 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -43,6 +43,9 @@
> do_rootfs() { -e
> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/configscript.sh|g' \ -e
> 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ -e
> 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
> +           -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
> +           -e
> 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' \
> +           -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>                ${WORKDIR}/multistrap.conf
>  
>      # Create root filesystem
> @@ -54,4 +57,4 @@ do_rootfs() {
>      sudo rm ${IMAGE_ROOTFS}/configscript.sh
>  }
>  
> -addtask rootfs before do_populate
> +addtask rootfs before do_build after do_populate
> diff --git a/meta/classes/ext4-img.bbclass
> b/meta/classes/ext4-img.bbclass index 3e3768b..9fa9aa5 100644
> --- a/meta/classes/ext4-img.bbclass
> +++ b/meta/classes/ext4-img.bbclass
> @@ -34,4 +34,4 @@ do_ext4_image() {
>      fi
>  }
>  
> -addtask ext4_image before do_build after do_populate
> +addtask ext4_image before do_build after do_rootfs
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index d59959b..410f30f 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -62,16 +62,6 @@ do_populate() {
>                       includedeb ${DEBDISTRONAME} \
>                       ${DEPLOY_DIR_DEB}/${p}_*.deb
>          done
> -
> -        sudo mkdir -p ${IMAGE_ROOTFS}/deb
> -
> -        for p in ${IMAGE_INSTALL}; do
> -            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
> -        done
> -
> -        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
> -
> -        sudo rm -rf ${IMAGE_ROOTFS}/deb
>      fi
>  }
>  

Re: [PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Henning Schild]

Am Wed, 20 Sep 2017 11:12:11 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 09/20/2017 10:58 AM, Henning Schild wrote:
> > Am Tue, 19 Sep 2017 15:20:49 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Add template for reprepro configuration file and function that
> >> generates final config for specific deistro.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta-isar-bin/conf/layer.conf        | 11 +++++++++++
> >>   meta-isar-bin/files/distributions.in |  3 +++
> >>   meta-isar/conf/bblayers.conf.sample  |  1 +
> >>   meta/classes/image.bbclass           | 25
> >> +++++++++++++++++++++++++ 4 files changed, 40 insertions(+)
> >>   create mode 100644 meta-isar-bin/conf/layer.conf
> >>   create mode 100644 meta-isar-bin/files/distributions.in
> >>
> >> diff --git a/meta-isar-bin/conf/layer.conf
> >> b/meta-isar-bin/conf/layer.conf new file mode 100644
> >> index 0000000..3518184
> >> --- /dev/null
> >> +++ b/meta-isar-bin/conf/layer.conf
> >> @@ -0,0 +1,11 @@
> >> +# This software is a part of ISAR.
> >> +# Copyright (C) 2017 ilbers GmbH
> >> +
> >> +# Codename of the repository created by the caching class
> >> +DEBDISTRONAME = "isar"
> >> +
> >> +# Path to the caching repository
> >> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
> >> +
> >> +# Path to the configuration files templates used by `reprepro`
> >> +DEBFILESDIR ?= "${LAYERDIR}/files"
> >> diff --git a/meta-isar-bin/files/distributions.in
> >> b/meta-isar-bin/files/distributions.in new file mode 100644
> >> index 0000000..cd214c6
> >> --- /dev/null
> >> +++ b/meta-isar-bin/files/distributions.in
> >> @@ -0,0 +1,3 @@
> >> +Codename: {DISTRO_NAME}
> >> +Architectures: i386 armhf amd64 source
> >> +Components: main
> >> diff --git a/meta-isar/conf/bblayers.conf.sample
> >> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
> >> --- a/meta-isar/conf/bblayers.conf.sample
> >> +++ b/meta-isar/conf/bblayers.conf.sample
> >> @@ -8,6 +8,7 @@ BBFILES ?= ""
> >>   BBLAYERS ?= " \
> >>     ##ISARROOT##/meta \
> >>     ##ISARROOT##/meta-isar \
> >> +  ##ISARROOT##/meta-isar-bin \
> >>     "
> >>   BBLAYERS_NON_REMOVABLE ?= " \
> >>     ##ISARROOT##/meta \
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index 5bf9524..30f241a 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
> >>   
> >>   inherit ${IMAGE_TYPE}
> >>   
> >> +# Generate apt cache configuration file
> >> +python __anonymous () {
> >> +    DISTRO = d.getVar("DISTRO", True)
> >> +    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
> >> +    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
> >> +
> >> +    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
> >> +    cache_conf_file = os.path.join(cache_conf_path,
> >> "distributions")
> >> +    cache_conf_file_in = os.path.join(DEBFILESDIR,
> >> "distributions.in") +
> >> +    import subprocess
> >> +
> >> +    if not os.path.exists(cache_conf_path):
> >> +        os.makedirs(cache_conf_path)
> >> +
> >> +    if not os.path.exists(cache_conf_file):
> >> +        f = open(cache_conf_file, "w")
> >> +        subprocess.check_call([
> >> +            "sed",
> >> +            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME",
> >> True) + "#g",
> >> +            cache_conf_file_in,
> >> +            ], stdout=f)
> >> +        f.close()
> >> +}  
> > 
> > Why is this not just a regular task early in the image chain?
> >   
> 
> The idea is to avoid races with multiconfig. If I add this as a
> regular task, it could be run in parallel by different multiconfigs,
> so there could occur races in `mkdir` and `sed`. The same is true
> when you try to build several images at the same time:
> 
>    $ bitbake image1 image2
>
> IIRC bitbake anonymous tasks are executed during parsing stage, so
> they are serialized.

I see, but that makes them a very strong measure that you can never
override i.e. in a layer on top. Not sure if that matters but i would
shy away from using the strongest measure if it can be done in a
different way.

For the dir how about using newtask[dirs] i guess bitbake will know how
to handle that and it seems to be the bitbake way anyways.

And for the sed, if the race happens and two or more seds get run. They
should just produce the same output and not fail if it was already there

sed "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME", True) + "#g" \
	cache_conf_file_in > cache_conf_file

Henning

> Alex
> 
> > Henning
> >   
> >>   do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
> >>   
> >>   # Install Debian packages, that were built from sources  
> >  

Re: [PATCH v2 3/4] meta-isar-bin: Populate cache

[Alexander Smirnov]

On 09/20/2017 11:22 AM, Henning Schild wrote:
> Am Tue, 19 Sep 2017 15:20:51 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Add newly built packages to apt cache.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/image.bbclass | 9 +++++++++
>>   1 file changed, 9 insertions(+)
>>
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index 9db7371..d59959b 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -54,6 +54,15 @@ do_populate[stamp-extra-info] =
>> "${DISTRO}-${MACHINE}" # Install Debian packages, that were built
>> from sources do_populate() {
>>       if [ -n "${IMAGE_INSTALL}" ]; then
>> +        for p in ${IMAGE_INSTALL}; do
>> +            reprepro --waitforlock 3 \
>> +                     -b ${DEBCACHEDIR}/${DISTRO} \
>> +                     --dbdir ${DEBDBDIR}/${DISTRO} \
>> +                     -C main \
>> +                     includedeb ${DEBDISTRONAME} \
>> +                     ${DEPLOY_DIR_DEB}/${p}_*.deb
>> +        done
> 
> Why not just do that in do_deploy_deb? Maybe DEPLOY_DIR_DEB could go
> away because we deploy directly into the repo.

This is an attempt to avoid races in reprepro, let me describe the usecase:

1. do_populate() is executed after do_build()

2. If we have several recipes for building, this means that 
do_populate() could be run in parallel (if you enable parallel jobs for 
bitbake)

3. --waitforlock does not guarantee that reprepro will take ownership 
after timeout

4. I agree that DEPLOY_DIR_DEB should go away, but at the moment I think 
we can use it for a while, until better solution will be found.

One approach could be walking through WORKDIRS:

for p in ${IMAGE_INSTALL}; do
     reprepro \
     ....
     ${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${p}-*/*.deb
done

but I'd like to think if some better solution could be found.

> 
> I found that for kernels built with "make deb-pkg" the "${p}_*.deb"
> pattern does not match. At least the way that i named the recipe.
> "do_deploy_deb" already works on WORKDIR/*.deb which naturally covers
> all packages coming out of a recipe, even if they had names different
> from the recipe. Imagine a recipe that downloads multiple .debs from
> somewhere and they all have very different names.

This works with ${WORKDIR}, where you know that is contains only 
packages you want (i.e. they are listed in ${IMAGE_INSTALL}). But deploy 
directory could contain 'unwanted' packages, that for example are left 
from previous builds. But I think we could ignore this case for now.

> 
> Again the question about the waitforlock, 3 seems only slightly better
> than 1. I think we need infinite or a serialization mechanism and 1.

Hmm, here you are right. If several architectures in multiconfig will 
populate ${DEPLOY_DIR_DEB} in parallel, 3 times won't be enough. Will 
try to implement loop to poll return value from reprepro without 
waitforlock.

Alex

> 
> Henning
> 
>>           sudo mkdir -p ${IMAGE_ROOTFS}/deb
>>   
>>           for p in ${IMAGE_INSTALL}; do
> 

Re: [PATCH v2 1/4] meta-isar-bin: Add reprepro configs

[Alexander Smirnov]

On 09/20/2017 11:38 AM, Henning Schild wrote:
> Am Wed, 20 Sep 2017 11:12:11 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 09/20/2017 10:58 AM, Henning Schild wrote:
>>> Am Tue, 19 Sep 2017 15:20:49 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Add template for reprepro configuration file and function that
>>>> generates final config for specific deistro.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    meta-isar-bin/conf/layer.conf        | 11 +++++++++++
>>>>    meta-isar-bin/files/distributions.in |  3 +++
>>>>    meta-isar/conf/bblayers.conf.sample  |  1 +
>>>>    meta/classes/image.bbclass           | 25
>>>> +++++++++++++++++++++++++ 4 files changed, 40 insertions(+)
>>>>    create mode 100644 meta-isar-bin/conf/layer.conf
>>>>    create mode 100644 meta-isar-bin/files/distributions.in
>>>>
>>>> diff --git a/meta-isar-bin/conf/layer.conf
>>>> b/meta-isar-bin/conf/layer.conf new file mode 100644
>>>> index 0000000..3518184
>>>> --- /dev/null
>>>> +++ b/meta-isar-bin/conf/layer.conf
>>>> @@ -0,0 +1,11 @@
>>>> +# This software is a part of ISAR.
>>>> +# Copyright (C) 2017 ilbers GmbH
>>>> +
>>>> +# Codename of the repository created by the caching class
>>>> +DEBDISTRONAME = "isar"
>>>> +
>>>> +# Path to the caching repository
>>>> +DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>>> +
>>>> +# Path to the configuration files templates used by `reprepro`
>>>> +DEBFILESDIR ?= "${LAYERDIR}/files"
>>>> diff --git a/meta-isar-bin/files/distributions.in
>>>> b/meta-isar-bin/files/distributions.in new file mode 100644
>>>> index 0000000..cd214c6
>>>> --- /dev/null
>>>> +++ b/meta-isar-bin/files/distributions.in
>>>> @@ -0,0 +1,3 @@
>>>> +Codename: {DISTRO_NAME}
>>>> +Architectures: i386 armhf amd64 source
>>>> +Components: main
>>>> diff --git a/meta-isar/conf/bblayers.conf.sample
>>>> b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644
>>>> --- a/meta-isar/conf/bblayers.conf.sample
>>>> +++ b/meta-isar/conf/bblayers.conf.sample
>>>> @@ -8,6 +8,7 @@ BBFILES ?= ""
>>>>    BBLAYERS ?= " \
>>>>      ##ISARROOT##/meta \
>>>>      ##ISARROOT##/meta-isar \
>>>> +  ##ISARROOT##/meta-isar-bin \
>>>>      "
>>>>    BBLAYERS_NON_REMOVABLE ?= " \
>>>>      ##ISARROOT##/meta \
>>>> diff --git a/meta/classes/image.bbclass
>>>> b/meta/classes/image.bbclass index 5bf9524..30f241a 100644
>>>> --- a/meta/classes/image.bbclass
>>>> +++ b/meta/classes/image.bbclass
>>>> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
>>>>    
>>>>    inherit ${IMAGE_TYPE}
>>>>    
>>>> +# Generate apt cache configuration file
>>>> +python __anonymous () {
>>>> +    DISTRO = d.getVar("DISTRO", True)
>>>> +    DEBFILESDIR = d.getVar("DEBFILESDIR", True)
>>>> +    DEBCACHEDIR = d.getVar("DEBCACHEDIR", True)
>>>> +
>>>> +    cache_conf_path = os.path.join(DEBCACHEDIR, DISTRO, "conf")
>>>> +    cache_conf_file = os.path.join(cache_conf_path,
>>>> "distributions")
>>>> +    cache_conf_file_in = os.path.join(DEBFILESDIR,
>>>> "distributions.in") +
>>>> +    import subprocess
>>>> +
>>>> +    if not os.path.exists(cache_conf_path):
>>>> +        os.makedirs(cache_conf_path)
>>>> +
>>>> +    if not os.path.exists(cache_conf_file):
>>>> +        f = open(cache_conf_file, "w")
>>>> +        subprocess.check_call([
>>>> +            "sed",
>>>> +            "-e", "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME",
>>>> True) + "#g",
>>>> +            cache_conf_file_in,
>>>> +            ], stdout=f)
>>>> +        f.close()
>>>> +}
>>>
>>> Why is this not just a regular task early in the image chain?
>>>    
>>
>> The idea is to avoid races with multiconfig. If I add this as a
>> regular task, it could be run in parallel by different multiconfigs,
>> so there could occur races in `mkdir` and `sed`. The same is true
>> when you try to build several images at the same time:
>>
>>     $ bitbake image1 image2
>>
>> IIRC bitbake anonymous tasks are executed during parsing stage, so
>> they are serialized.
> 
> I see, but that makes them a very strong measure that you can never
> override i.e. in a layer on top. Not sure if that matters but i would
> shy away from using the strongest measure if it can be done in a
> different way.

Agree.

> 
> For the dir how about using newtask[dirs] i guess bitbake will know how
> to handle that and it seems to be the bitbake way anyways.

Let me try :-)

> 
> And for the sed, if the race happens and two or more seds get run. They
> should just produce the same output and not fail if it was already there
> 
> sed "s#{DISTRO_NAME}#" + d.getVar("DEBDISTRONAME", True) + "#g" \
> 	cache_conf_file_in > cache_conf_file
> 

Ok.

> Henning
> 
>> Alex
>>
>>> Henning
>>>    
>>>>    do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
>>>>    
>>>>    # Install Debian packages, that were built from sources
>>>   
>

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Andreas Reichel]

On Wed, Sep 20, 2017 at 11:26:20AM +0300, Alexander Smirnov wrote:
> 
> After build, there will be the following in the cache (example for hello):
> 
> $ ls -1 meta-isar-bin/apt/debian-jessie/pool/main/h/hello/
> hello_0.1+g7f35942-1_amd64.deb
> hello_0.1+g7f35942-1_armhf.deb
> hello_0.1+g7f35942-1_i386.deb
> 

This looks like a misdesign, because meta-layers should not be populated
by the build process and furthermore, should contain recipes and not a
cache itself. It would be like the sstate-cache lying inside meta-oe...

Why not define a variable like 'DEB_CACHE_DIR' or something alike and
use this directory. Then you also have more flexibility and can handle
multiple different caches, i.e. the caches become selectable and
independent of the layer itself...

Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
snippents for multiconf and then you can have separate caches for
different target architectures and don't mix everything up in one pool.

This way you can easily drop a pool by architecture without sorting
files out.

Kind regards
Andreas

-- 
Andreas Reichel
Dipl.-Phys. (Univ.)
Software Consultant

Andreas.Reichel@tngtech.com, +49-174-3180074
TNG Technology Consulting GmbH, Betastr. 13a, 85774 Unterfoehring
Geschaeftsfuehrer: Henrik Klagges, Dr. Robert Dahlke, Gerhard Mueller
Sitz: Unterfoehring * Amtsgericht Muenchen * HRB 135082

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Claudius Heine]

Hi,

On 09/21/2017 10:55 AM, [ext] Andreas Reichel wrote:
> On Wed, Sep 20, 2017 at 11:26:20AM +0300, Alexander Smirnov wrote:
>>
>> After build, there will be the following in the cache (example for hello):
>>
>> $ ls -1 meta-isar-bin/apt/debian-jessie/pool/main/h/hello/
>> hello_0.1+g7f35942-1_amd64.deb
>> hello_0.1+g7f35942-1_armhf.deb
>> hello_0.1+g7f35942-1_i386.deb
>>
> 
> This looks like a misdesign, because meta-layers should not be populated
> by the build process and furthermore, should contain recipes and not a
> cache itself. It would be like the sstate-cache lying inside meta-oe...
> 
> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
> use this directory. Then you also have more flexibility and can handle
> multiple different caches, i.e. the caches become selectable and
> independent of the layer itself...
> 
> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
> snippents for multiconf and then you can have separate caches for
> different target architectures and don't mix everything up in one pool.
> 
> This way you can easily drop a pool by architecture without sorting
> files out.

This patch is sort of doing that. It defines:

     DEBDISTRONAME = "isar"
     DEBCACHEDIR ?= "${LAYERDIR}/apt"
     DEBFILESDIR ?= "${LAYERDIR}/files"

In the layer.conf of the 'meta-isar-bin'. So you could overwrite it in 
the local.conf. But I agree that this way of doing thing looks rather 
strange, because now we have a layer that does not contain meta data 
(recipes, patches, etc.) but instead it contains binary packages.

The idea of layers is to have multiple of them laying over each other 
creating some kind of union file system. You can't do that here AFAIK. 
So it goes against pretty much the basic concept of bitbake and the 
layering.

Also putting those build products in the meta source tree is very bold. 
I could imagine that those meta layers are mounted as read only into the 
virtual machine/container. This would pretty much break this.

So I would be in favor of handling this kind of binary cache similar to 
how bitbake handles the the 'DL_DIR' and 'SSTATE_DIR'.

Cheers,
Claudius


-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Baurzhan Ismagulov]

Hello Andreas and Claudius,

On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
> This looks like a misdesign, because meta-layers should not be populated
> by the build process and furthermore, should contain recipes and not a
> cache itself. It would be like the sstate-cache lying inside meta-oe...
> 
> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
> use this directory. Then you also have more flexibility and can handle
> multiple different caches, i.e. the caches become selectable and
> independent of the layer itself...
> 
> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
> snippents for multiconf and then you can have separate caches for
> different target architectures and don't mix everything up in one pool.
> 
> This way you can easily drop a pool by architecture without sorting
> files out.

I share your concerns and don't like that myself. That said, it's a demo
implementation we ended up considering the following project requirements:

R1. Product's binary repo is persistent across builds on the same host and
    across the hosts of different developers.

R2. Building isar-image-base should be kept simple. New users should not be
    overwhelmed with repo setup, external tools, or manual configuration (as
    much as possible).

R3. Build directory must be removable at any time without affecting cloned or
    newly built binary cache.

We've translated that to the following design points:

D1. Apt repo is versioned, e.g. in git.

D2. Config files for this feature are put under isar/...

D3. Given D1 and D2, binary packages go under isar/... :( .

The idea is that in a project, the cache is a separate repo at the same level
as Isar:

product (originally cloned from isar)
product-bin

So, for production use, a tool like kas should be used to clone both at right
revisions. We didn't want to do that for the demo due to R2. Just like
meta-isar is currently a template for real products, meta-isar-bin is an
example that should be moved out of the product source repo for a real product.

Technically, meta-isar-bin is a layer that contains configuration files that
have to be put somewhere. As package installation via apt is a requirement
(dpkg doesn't install package dependencies), meta-isar-bin should become the
default and would thus be required in the current implementation.

We had considered cloning meta-isar-bin into the build directory, but that
would mean manual configuration for new users, and (possibly modified) configs
could be deleted if the build directory is deleted -- an unpleasant surprise
for users familiar with OE.

If it's a separate directory, the config files should go there, since they are
per-repo, and there could be several ones:

product
company-bin
department-bin
product-bin

For a source distribution, having binary repos as layers sounds perverse. But
given Isar's focus on binary packages, it's layering in the sense that more
specific repos could override more general ones.

In the future, we'd like to move more stuff into the core Isar (either moving
files from meta-isar to meta, or merging meta into meta-isar and introducing
meta-template). Maybe we could introduce kas in a simple way, which would solve
this problem.

So, ATM I'd suggest to document the steps regarding meta-isar-bin for creating
real products.

If anyone has an elegant solution for the issues above, I would be glad to hear
it. I agree with a separate directory approach, but suggest to postpone it till
we find a good way to introduce it to new users.

With kind regards,
Baurzhan.

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Claudius Heine]

Hi Baurzhan,

On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote:
> Hello Andreas and Claudius,
> 
> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
>> This looks like a misdesign, because meta-layers should not be populated
>> by the build process and furthermore, should contain recipes and not a
>> cache itself. It would be like the sstate-cache lying inside meta-oe...
>>
>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
>> use this directory. Then you also have more flexibility and can handle
>> multiple different caches, i.e. the caches become selectable and
>> independent of the layer itself...
>>
>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
>> snippents for multiconf and then you can have separate caches for
>> different target architectures and don't mix everything up in one pool.
>>
>> This way you can easily drop a pool by architecture without sorting
>> files out.
> 
> I share your concerns and don't like that myself. That said, it's a demo
> implementation we ended up considering the following project requirements:
> 
> R1. Product's binary repo is persistent across builds on the same host and
>      across the hosts of different developers.

Why does the binary repo needs to be be 'persistent' across different hosts?

This repo is only about the packages that are created by the recipes and 
those should already build reproducible packages so they already are 
'persistent'. So why do you need a cache that can be shared with other 
hosts?

Creating this sharable cache just raises the problem that some packages 
aren't build all the time. So its possible that if they break at some 
later point, it might happen unnoticed, because all devs and maybe even 
the CI is using some binary cache. Not ideal.

You have to think about what is the real point of this cache is.

Normally a build cache is there to make succeeding builds faster. To 
detect if some item in the cache needs to be deleted, because the source 
has updated is not a trivial problem. To do that with multiple hosts 
involved it becomes much more difficult.

If you put the cache into a version control system, you have to bind the 
cache to the exact revisions of the project repositories. Just to have 
some knowledge about how those packages are created. But those revisions 
hashes might be unique, but they are not persistent. The can disappear 
with a rebase, etc. And now we get into the situations where we have a 
cache that references some non existing repo revisions and nobody knows 
how the packages are build, but since they do exactly what is needed, 
nobody touches the sources anymore, but instead start to modify the 
cached packages directly if they want small changes. That is the 
nightmare scenario for reproducible builds.

The longer I think about this, the more bad scenarios I can think of. 
And I am sure the reality might teach us all some more in time.
Please don't give the developers a repository with binary packages, they 
really should not be exposed to this kind of seduction.

> R2. Building isar-image-base should be kept simple. New users should not be
>      overwhelmed with repo setup, external tools, or manual configuration (as
>      much as possible).
> 
> R3. Build directory must be removable at any time without affecting cloned or
>      newly built binary cache.
> 
> We've translated that to the following design points:
> 
> D1. Apt repo is versioned, e.g. in git.
> 
> D2. Config files for this feature are put under isar/...
> 
> D3. Given D1 and D2, binary packages go under isar/... :( .
> 
> The idea is that in a project, the cache is a separate repo at the same level
> as Isar:
> 
> product (originally cloned from isar)
> product-bin
> 
> So, for production use, a tool like kas should be used to clone both at right
> revisions. We didn't want to do that for the demo due to R2. Just like
> meta-isar is currently a template for real products, meta-isar-bin is an
> example that should be moved out of the product source repo for a real product.

Since you mention the 'meta' and 'meta-isar' layers. I would be in favor 
of moving everything from the 'meta' layer into the 'meta-isar' layer. 
We don't really need two layers IMO. We could put examples into 
'meta-isar-examples'.

Maybe we should discuss this separately.

> 
> Technically, meta-isar-bin is a layer that contains configuration files that
> have to be put somewhere. As package installation via apt is a requirement
> (dpkg doesn't install package dependencies), meta-isar-bin should become the
> default and would thus be required in the current implementation.
> 
> We had considered cloning meta-isar-bin into the build directory, but that
> would mean manual configuration for new users, and (possibly modified) configs
> could be deleted if the build directory is deleted -- an unpleasant surprise
> for users familiar with OE.
> 
> If it's a separate directory, the config files should go there, since they are
> per-repo, and there could be several ones:
> 
> product
> company-bin
> department-bin
> product-bin
> 
> For a source distribution, having binary repos as layers sounds perverse. But
> given Isar's focus on binary packages, it's layering in the sense that more
> specific repos could override more general ones.
> 
> In the future, we'd like to move more stuff into the core Isar (either moving
> files from meta-isar to meta, or merging meta into meta-isar and introducing
> meta-template). Maybe we could introduce kas in a simple way, which would solve
> this problem.
> 
> So, ATM I'd suggest to document the steps regarding meta-isar-bin for creating
> real products.
> 
> If anyone has an elegant solution for the issues above, I would be glad to hear
> it. I agree with a separate directory approach, but suggest to postpone it till
> we find a good way to introduce it to new users.

I am still not convinced about the need to put binary repositories into 
layers. And introducing just a simple binary package repository as a 
package cache in the build directory does not need any special 
configuration files or overwriting existing configuration AFAIK. So 
introducing it to 'new users' would just work transparent.

Of course my solution would be a host local cache. But as I said, I 
don't really see any reason for having some kind of project global cache 
that is shared over multiple hosts. Ok you have a first first build, but
at the same time you don't even test if those packages can still be 
created from the sources/recipes. So thats not really an argument for me.

Cheers,
Claudius



-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov]

Hi,

On 09/25/2017 01:49 PM, Claudius Heine wrote:
> Hi Baurzhan,
> 
> On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote:
>> Hello Andreas and Claudius,
>>
>> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
>>> This looks like a misdesign, because meta-layers should not be populated
>>> by the build process and furthermore, should contain recipes and not a
>>> cache itself. It would be like the sstate-cache lying inside meta-oe...
>>>
>>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
>>> use this directory. Then you also have more flexibility and can handle
>>> multiple different caches, i.e. the caches become selectable and
>>> independent of the layer itself...
>>>
>>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
>>> snippents for multiconf and then you can have separate caches for
>>> different target architectures and don't mix everything up in one pool.
>>>
>>> This way you can easily drop a pool by architecture without sorting
>>> files out.
>>
>> I share your concerns and don't like that myself. That said, it's a demo
>> implementation we ended up considering the following project 
>> requirements:
>>
>> R1. Product's binary repo is persistent across builds on the same host 
>> and
>>      across the hosts of different developers.
> 
> Why does the binary repo needs to be be 'persistent' across different 
> hosts?

In general Isar designed to work with binary repositories, that's the 
key feature.

Nobody forces you to use this cache persistent across different hosts, 
you are able to build everything from sources at any time.
But at the same time I want to have the possibility to cache some stable 
Isar part for my needs and do not rebuild it everytime.

> 
> This repo is only about the packages that are created by the recipes and 
> those should already build reproducible packages so they already are 
> 'persistent'. So why do you need a cache that can be shared with other 
> hosts?

This topic is not about reproducible builds. Each software product has 
sources, so it can be built from the scratch at anytime. But why I need 
to do this, when I have binary? Installing Debian, user usually prefers 
binary apt instead of building the system from sources. It's just more 
comfortable and takes much less time.

Also having Isar in binary repo in general I don't need the Isar at all 
(including setup development host). I'm able to generate target 
filesystem by using multistrap only.

> 
> Creating this sharable cache just raises the problem that some packages 
> aren't build all the time. So its possible that if they break at some 
> later point, it might happen unnoticed, because all devs and maybe even 
> the CI is using some binary cache. Not ideal.

This problem doesn't relate to binary cache. If the software package 
contains a bug, it doesn't matter if the package is binary or it's built 
from sources every time. If after bug-fix the respective binary package 
wasn't updated - it's an issue of engineer. If you don't want to export 
the cache - you don't need to think about this problem.

> 
> You have to think about what is the real point of this cache is.
> 
> Normally a build cache is there to make succeeding builds faster. To 
> detect if some item in the cache needs to be deleted, because the source 
> has updated is not a trivial problem. To do that with multiple hosts 
> involved it becomes much more difficult.

Actually it doesn't. The cache is built using reprepro, which provides 
possibility to query various information for specific packages. So, for 
example, it's quite easy to implement the following steps in anonymous 
function:
  - get binary package version
  - compare with the version in recipe
  - if it differs - force Isar to update the cache for this package

> 
> If you put the cache into a version control system, you have to bind the 
> cache to the exact revisions of the project repositories. Just to have 
> some knowledge about how those packages are created. But those revisions 
> hashes might be unique, but they are not persistent. The can disappear 
> with a rebase, etc. And now we get into the situations where we have a 
> cache that references some non existing repo revisions and nobody knows 
> how the packages are build, but since they do exactly what is needed, 
> nobody touches the sources anymore, but instead start to modify the 
> cached packages directly if they want small changes. That is the 
> nightmare scenario for reproducible builds.
> 
> The longer I think about this, the more bad scenarios I can think of. 
> And I am sure the reality might teach us all some more in time.
> Please don't give the developers a repository with binary packages, they 
> really should not be exposed to this kind of seduction.

I don't agree here, it's a matter of taste. Again, nobody forces you to 
export the cache. For me - I don't want to spend hours building 
everything from sources, when I know that there are stable binaries.

> 
>> R2. Building isar-image-base should be kept simple. New users should 
>> not be
>>      overwhelmed with repo setup, external tools, or manual 
>> configuration (as
>>      much as possible).
>>
>> R3. Build directory must be removable at any time without affecting 
>> cloned or
>>      newly built binary cache.
>>
>> We've translated that to the following design points:
>>
>> D1. Apt repo is versioned, e.g. in git.
>>
>> D2. Config files for this feature are put under isar/...
>>
>> D3. Given D1 and D2, binary packages go under isar/... :( .
>>
>> The idea is that in a project, the cache is a separate repo at the 
>> same level
>> as Isar:
>>
>> product (originally cloned from isar)
>> product-bin
>>
>> So, for production use, a tool like kas should be used to clone both 
>> at right
>> revisions. We didn't want to do that for the demo due to R2. Just like
>> meta-isar is currently a template for real products, meta-isar-bin is an
>> example that should be moved out of the product source repo for a real 
>> product.
> 
> Since you mention the 'meta' and 'meta-isar' layers. I would be in favor 
> of moving everything from the 'meta' layer into the 'meta-isar' layer. 
> We don't really need two layers IMO. We could put examples into 
> 'meta-isar-examples'.
> 
> Maybe we should discuss this separately.

Yes, I'd suggest to discuss this out of this thread.

> 
>>
>> Technically, meta-isar-bin is a layer that contains configuration 
>> files that
>> have to be put somewhere. As package installation via apt is a 
>> requirement
>> (dpkg doesn't install package dependencies), meta-isar-bin should 
>> become the
>> default and would thus be required in the current implementation.
>>
>> We had considered cloning meta-isar-bin into the build directory, but 
>> that
>> would mean manual configuration for new users, and (possibly modified) 
>> configs
>> could be deleted if the build directory is deleted -- an unpleasant 
>> surprise
>> for users familiar with OE.
>>
>> If it's a separate directory, the config files should go there, since 
>> they are
>> per-repo, and there could be several ones:
>>
>> product
>> company-bin
>> department-bin
>> product-bin
>>
>> For a source distribution, having binary repos as layers sounds 
>> perverse. But
>> given Isar's focus on binary packages, it's layering in the sense that 
>> more
>> specific repos could override more general ones.
>>
>> In the future, we'd like to move more stuff into the core Isar (either 
>> moving
>> files from meta-isar to meta, or merging meta into meta-isar and 
>> introducing
>> meta-template). Maybe we could introduce kas in a simple way, which 
>> would solve
>> this problem.
>>
>> So, ATM I'd suggest to document the steps regarding meta-isar-bin for 
>> creating
>> real products.
>>
>> If anyone has an elegant solution for the issues above, I would be 
>> glad to hear
>> it. I agree with a separate directory approach, but suggest to 
>> postpone it till
>> we find a good way to introduce it to new users.
> 
> I am still not convinced about the need to put binary repositories into 
> layers. And introducing just a simple binary package repository as a 
> package cache in the build directory does not need any special 
> configuration files or overwriting existing configuration AFAIK. So 
> introducing it to 'new users' would just work transparent.

reprepro requires configuration file.

> 
> Of course my solution would be a host local cache. But as I said, I 
> don't really see any reason for having some kind of project global cache 
> that is shared over multiple hosts. Ok you have a first first build, but
> at the same time you don't even test if those packages can still be 
> created from the sources/recipes. So thats not really an argument for me.
> 

But Debian users prefers to use ftp://ftp.debian.org/debian/ instead of 
building from tarballs.

-- 
With best regards,
Alexander Smirnov

Re: [PATCH v2 2/4] meta-isar-bin: Generate cache repos

[Claudius Heine]

Hi Alex,

On 09/25/2017 01:57 PM, Alexander Smirnov wrote:
> Hi,
> 
> On 09/25/2017 01:49 PM, Claudius Heine wrote:
>> Hi Baurzhan,
>>
>> On 09/22/2017 12:56 PM, Baurzhan Ismagulov wrote:
>>> Hello Andreas and Claudius,
>>>
>>> On Thu, Sep 21, 2017 at 10:55:37AM +0200, Andreas Reichel wrote:
>>>> This looks like a misdesign, because meta-layers should not be 
>>>> populated
>>>> by the build process and furthermore, should contain recipes and not a
>>>> cache itself. It would be like the sstate-cache lying inside meta-oe...
>>>>
>>>> Why not define a variable like 'DEB_CACHE_DIR' or something alike and
>>>> use this directory. Then you also have more flexibility and can handle
>>>> multiple different caches, i.e. the caches become selectable and
>>>> independent of the layer itself...
>>>>
>>>> Furthermore, you could set the DEB_CACHE_DIR variable in the local.conf
>>>> snippents for multiconf and then you can have separate caches for
>>>> different target architectures and don't mix everything up in one pool.
>>>>
>>>> This way you can easily drop a pool by architecture without sorting
>>>> files out.
>>>
>>> I share your concerns and don't like that myself. That said, it's a demo
>>> implementation we ended up considering the following project 
>>> requirements:
>>>
>>> R1. Product's binary repo is persistent across builds on the same 
>>> host and
>>>      across the hosts of different developers.
>>
>> Why does the binary repo needs to be be 'persistent' across different 
>> hosts?
> 
> In general Isar designed to work with binary repositories, that's the 
> key feature.
> 
> Nobody forces you to use this cache persistent across different hosts, 
> you are able to build everything from sources at any time.
> But at the same time I want to have the possibility to cache some stable 
> Isar part for my needs and do not rebuild it everytime.

Yes, that is understandable. I don't dispute the usefulness of a cache 
in general, but the usefulness of a shareable cache.

IMO a cache is local to the build host and should not be shared.

> 
>>
>> This repo is only about the packages that are created by the recipes 
>> and those should already build reproducible packages so they already 
>> are 'persistent'. So why do you need a cache that can be shared with 
>> other hosts?
> 
> This topic is not about reproducible builds. Each software product has 
> sources, so it can be built from the scratch at anytime. But why I need 
> to do this, when I have binary? Installing Debian, user usually prefers 
> binary apt instead of building the system from sources. It's just more 
> comfortable and takes much less time.

The end user would rather have just a binary image to download and use. 
That takes even less time and is just slightly less comfortable than 
buying a micro sd card with the image already flashed on it.

> 
> Also having Isar in binary repo in general I don't need the Isar at all 
> (including setup development host). I'm able to generate target 
> filesystem by using multistrap only.

Then why make this part of isar if you intent to use it without isar?

The point is, if some user uses isar to create the image, she wants to 
build the image from, what isar considers, 'source' files. Those source 
files might be binary packages or source code that is compiled and 
packaged by isar.

What you are doing here is extracting some intermediate product of isar 
and planning to distribute it, and that is a really bad idea. Its 
against principles of reproducibility and maintainability.

As I already described, at some point in the future. Some developer will 
take this intermediate product of isar, change it and put it back into 
the isar image creation process. Then you get a desyncronisation between 
what isar considers 'source' and the intermediate product the developer 
has just modified.

>> Creating this sharable cache just raises the problem that some 
>> packages aren't build all the time. So its possible that if they break 
>> at some later point, it might happen unnoticed, because all devs and 
>> maybe even the CI is using some binary cache. Not ideal.
> 
> This problem doesn't relate to binary cache. If the software package 
> contains a bug, it doesn't matter if the package is binary or it's built 
> from sources every time. If after bug-fix the respective binary package 
> wasn't updated - it's an issue of engineer. If you don't want to export 
> the cache - you don't need to think about this problem.

My point it that exporting the cache should not be a requirement, 
because its a bad idea that might cause may problems.

> 
>>
>> You have to think about what is the real point of this cache is.
>>
>> Normally a build cache is there to make succeeding builds faster. To 
>> detect if some item in the cache needs to be deleted, because the 
>> source has updated is not a trivial problem. To do that with multiple 
>> hosts involved it becomes much more difficult.
> 
> Actually it doesn't. The cache is built using reprepro, which provides 
> possibility to query various information for specific packages. So, for 
> example, it's quite easy to implement the following steps in anonymous 
> function:
>   - get binary package version
>   - compare with the version in recipe
>   - if it differs - force Isar to update the cache for this package

And thats wrong. For instance I choose to change the install path of a 
binary from /usr/bin to /bin. I wouldn't need to change the version of 
the package, because it still does the same. With our mechanism it 
wouldn't rebuild because the version is the same. Bitbake can already 
handle these kind of issues.

> 
>>
>> If you put the cache into a version control system, you have to bind 
>> the cache to the exact revisions of the project repositories. Just to 
>> have some knowledge about how those packages are created. But those 
>> revisions hashes might be unique, but they are not persistent. The can 
>> disappear with a rebase, etc. And now we get into the situations where 
>> we have a cache that references some non existing repo revisions and 
>> nobody knows how the packages are build, but since they do exactly 
>> what is needed, nobody touches the sources anymore, but instead start 
>> to modify the cached packages directly if they want small changes. 
>> That is the nightmare scenario for reproducible builds.
>>
>> The longer I think about this, the more bad scenarios I can think of. 
>> And I am sure the reality might teach us all some more in time.
>> Please don't give the developers a repository with binary packages, 
>> they really should not be exposed to this kind of seduction.
> 
> I don't agree here, it's a matter of taste. Again, nobody forces you to 
> export the cache. For me - I don't want to spend hours building 
> everything from sources, when I know that there are stable binaries.

As I said, you are missing my point. I have nothing against caches in 
general. As long as they stay local to the build host.

>>
>>>
>>> Technically, meta-isar-bin is a layer that contains configuration 
>>> files that
>>> have to be put somewhere. As package installation via apt is a 
>>> requirement
>>> (dpkg doesn't install package dependencies), meta-isar-bin should 
>>> become the
>>> default and would thus be required in the current implementation.
>>>
>>> We had considered cloning meta-isar-bin into the build directory, but 
>>> that
>>> would mean manual configuration for new users, and (possibly 
>>> modified) configs
>>> could be deleted if the build directory is deleted -- an unpleasant 
>>> surprise
>>> for users familiar with OE.
>>>
>>> If it's a separate directory, the config files should go there, since 
>>> they are
>>> per-repo, and there could be several ones:
>>>
>>> product
>>> company-bin
>>> department-bin
>>> product-bin
>>>
>>> For a source distribution, having binary repos as layers sounds 
>>> perverse. But
>>> given Isar's focus on binary packages, it's layering in the sense 
>>> that more
>>> specific repos could override more general ones.
>>>
>>> In the future, we'd like to move more stuff into the core Isar 
>>> (either moving
>>> files from meta-isar to meta, or merging meta into meta-isar and 
>>> introducing
>>> meta-template). Maybe we could introduce kas in a simple way, which 
>>> would solve
>>> this problem.
>>>
>>> So, ATM I'd suggest to document the steps regarding meta-isar-bin for 
>>> creating
>>> real products.
>>>
>>> If anyone has an elegant solution for the issues above, I would be 
>>> glad to hear
>>> it. I agree with a separate directory approach, but suggest to 
>>> postpone it till
>>> we find a good way to introduce it to new users.
>>
>> I am still not convinced about the need to put binary repositories 
>> into layers. And introducing just a simple binary package repository 
>> as a package cache in the build directory does not need any special 
>> configuration files or overwriting existing configuration AFAIK. So 
>> introducing it to 'new users' would just work transparent.
> 
> reprepro requires configuration file.

So does wic. But wic does not need a separate layer just to contain its 
files.

> 
>>
>> Of course my solution would be a host local cache. But as I said, I 
>> don't really see any reason for having some kind of project global 
>> cache that is shared over multiple hosts. Ok you have a first first 
>> build, but
>> at the same time you don't even test if those packages can still be 
>> created from the sources/recipes. So thats not really an argument for me.
>>
> 
> But Debian users prefers to use ftp://ftp.debian.org/debian/ instead of 
> building from tarballs.

What does Debian users have to do with this? I would suggest we should 
target isar users and those are mostly developers that want to create 
Debian based images. Those developers what their own software deployed 
to a target machine and I expect that their own software is not 
available of ftp.debian.org.
So I don't get this point.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de