[PATCH v4 0/4] Basic binary cache implementation

[PATCH v4 0/4] Basic binary cache implementation

[Alexander Smirnov]

Hello everybody,

this series introduces basic binary caching for Isar.

There is a new layer: meta-isar-bin which is intended to be the binary
cache. All the packages that are built in Isar are stored in this
cache using reprepro.

Having the separate layer makes possible to manage this cache separately
from the Isar build tree. So once you have built your packages, you could
re-use this cache for further builds.

Also this cache repo is now used to generate image filesystem via
multistrap.

Changes since v3:
 - Check return code from reprepro, continue loop only if EEXIST code is
   returned. Othrwise build is aborted.

Changes since v2:
 - Droped anonymous functions
 - Added number of attempts to access apt cache before fail

Changes since v1:
 - Total refactoring
 - Split huge patches into small pieces
 - Avoid races

With best regards,
Alex

Alexander Smirnov (4):
  meta-isar-bin: Add reprepro configs
  meta-isar-bin: Generate cache repos
  meta-isar-bin: Populate cache
  meta-isar-bin: Install packages via multistrap

 meta-isar-bin/conf/layer.conf                      | 14 +++++
 meta-isar-bin/files/distributions.in               |  3 ++
 meta-isar/conf/bblayers.conf.sample                |  1 +
 meta-isar/conf/local.conf.sample                   |  4 ++
 .../recipes-core/images/files/multistrap.conf.in   |  9 +++-
 meta-isar/recipes-core/images/isar-image-base.bb   |  5 +-
 meta/classes/ext4-img.bbclass                      |  2 +-
 meta/classes/image.bbclass                         | 62 +++++++++++++++++++---
 8 files changed, 89 insertions(+), 11 deletions(-)
 create mode 100644 meta-isar-bin/conf/layer.conf
 create mode 100644 meta-isar-bin/files/distributions.in

-- 
2.1.4

[PATCH 1/4] meta-isar-bin: Add reprepro configs

[Alexander Smirnov]

Add template for reprepro configuration file and function that
generates final config for specific deistro.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar-bin/conf/layer.conf        | 11 +++++++++++
 meta-isar-bin/files/distributions.in |  3 +++
 meta-isar/conf/bblayers.conf.sample  |  1 +
 meta/classes/image.bbclass           | 15 +++++++++++++++
 4 files changed, 30 insertions(+)
 create mode 100644 meta-isar-bin/conf/layer.conf
 create mode 100644 meta-isar-bin/files/distributions.in

diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
new file mode 100644
index 0000000..3518184
--- /dev/null
+++ b/meta-isar-bin/conf/layer.conf
@@ -0,0 +1,11 @@
+# This software is a part of ISAR.
+# Copyright (C) 2017 ilbers GmbH
+
+# Codename of the repository created by the caching class
+DEBDISTRONAME = "isar"
+
+# Path to the caching repository
+DEBCACHEDIR ?= "${LAYERDIR}/apt"
+
+# Path to the configuration files templates used by `reprepro`
+DEBFILESDIR ?= "${LAYERDIR}/files"
diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta-isar-bin/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta-isar/conf/bblayers.conf.sample b/meta-isar/conf/bblayers.conf.sample
index 80867e7..53a362b 100644
--- a/meta-isar/conf/bblayers.conf.sample
+++ b/meta-isar/conf/bblayers.conf.sample
@@ -8,6 +8,7 @@ BBFILES ?= ""
 BBLAYERS ?= " \
   ##ISARROOT##/meta \
   ##ISARROOT##/meta-isar \
+  ##ISARROOT##/meta-isar-bin \
   "
 BBLAYERS_NON_REMOVABLE ?= " \
   ##ISARROOT##/meta \
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5bf9524..d30b139 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -10,6 +10,21 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
 
 inherit ${IMAGE_TYPE}
 
+CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+    if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+        sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
+            ${DEBFILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+    fi
+}
+
+addtask cache_config before do_fetch
+
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 
 # Install Debian packages, that were built from sources
-- 
2.1.4

[PATCH 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov]

Generate repos for each configuration in multiconfig.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar-bin/conf/layer.conf    |  3 +++
 meta-isar/conf/local.conf.sample |  4 ++++
 meta/classes/image.bbclass       | 35 ++++++++++++++++++++++++++++++++++-
 3 files changed, 41 insertions(+), 1 deletion(-)

diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf
index 3518184..5ef6e60 100644
--- a/meta-isar-bin/conf/layer.conf
+++ b/meta-isar-bin/conf/layer.conf
@@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
 
 # Path to the configuration files templates used by `reprepro`
 DEBFILESDIR ?= "${LAYERDIR}/files"
+
+# Path to the databases used by `reprepro`
+DEBDBDIR ?= "${LAYERDIR}/db"
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index a456b1b..660958f 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
 #
 # Default parallel jobs for bitbake:
 BB_NUMBER_THREADS = "4"
+
+#
+# Number of attempts to try to get reprepro lock for access to apt cache
+REPREPRO_LOCK_ATTEMPTS = "16"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index d30b139..b07775a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,5 +1,5 @@
 # This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
+# Copyright (C) 2015-2017 ilbers GmbH
 
 KERNEL_IMAGE ?= ""
 INITRD_IMAGE ?= ""
@@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
 do_cache_config[dirs] = "${CACHE_CONF_DIR}"
 do_cache_config[stamp-extra-info] = "${DISTRO}"
 
+call_reprepro() {
+    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
+        #  According to `sh` manual page, shell exit statuses fall between
+        # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
+        # integer results value (239).
+        eexist="239"
+        retval="0"
+        reprepro --waitforlock 1 $* || retval="$?"
+
+        # If reprepro has failed to get database lock, it returns EEXIST code.
+        # In this case we continue trying to get lock until max amount of
+        # attempts is reached.
+        if [ $retval -eq $eexist ]; then
+            bbwarn "Failed to get reprepro lock, trying again..."
+        else
+            break
+        fi
+    done
+
+    if [ $retval -ne 0 ]; then
+        bbfatal "reprepro failed"
+    fi
+}
+
 # Generate reprepro config for current distro if it doesn't exist. Once it's
 # generated, this task should do nothing.
 do_cache_config() {
@@ -21,6 +45,15 @@ do_cache_config() {
         sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
             ${DEBFILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
     fi
+
+    path_cache="${DEBCACHEDIR}/${DISTRO}"
+    path_databases="${DEBDBDIR}/${DISTRO}"
+
+    if [ ! -d "${path_databases}" ]; then
+        call_reprepro -b ${path_cache} \
+                      --dbdir ${path_databases} \
+                      export ${DEBDISTRONAME}
+    fi
 }
 
 addtask cache_config before do_fetch
-- 
2.1.4

[PATCH 3/4] meta-isar-bin: Populate cache

[Alexander Smirnov]

Add newly built packages to apt cache.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/image.bbclass | 8 ++++++++
 1 file changed, 8 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index b07775a..cf819e6 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -63,6 +63,14 @@ do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 # Install Debian packages, that were built from sources
 do_populate() {
     if [ -n "${IMAGE_INSTALL}" ]; then
+        for p in ${IMAGE_INSTALL}; do
+            call_reprepro -b ${DEBCACHEDIR}/${DISTRO} \
+                          --dbdir ${DEBDBDIR}/${DISTRO} \
+                          -C main \
+                          includedeb ${DEBDISTRONAME} \
+                          ${DEPLOY_DIR_DEB}/${p}_*.deb
+        done
+
         sudo mkdir -p ${IMAGE_ROOTFS}/deb
 
         for p in ${IMAGE_INSTALL}; do
-- 
2.1.4

[PATCH 4/4] meta-isar-bin: Install packages via multistrap

[Alexander Smirnov]

Install custom packages to target image filesystem using multistrap.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
 meta-isar/recipes-core/images/isar-image-base.bb       |  5 ++++-
 meta/classes/ext4-img.bbclass                          |  2 +-
 meta/classes/image.bbclass                             | 10 ----------
 4 files changed, 13 insertions(+), 13 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..38d53cf 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
 [General]
 noauth=true
 unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
 aptsources=##DISTRO##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
 components=##DISTRO_COMPONENTS##
 packages=##IMAGE_PREINSTALL##
 omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_DEB##
+suite=##ISAR_DISTRO_SUITE##
+components=main
+omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index f4b0125..3b29a53 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -46,6 +46,9 @@ do_rootfs() {
         -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
         -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+        -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' \
+        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
     # Create root filesystem
@@ -57,4 +60,4 @@ do_rootfs() {
     sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
 }
 
-addtask rootfs before do_populate after do_unpack
+addtask rootfs before do_build after do_populate
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
     fi
 }
 
-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index cf819e6..ff8a370 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -70,16 +70,6 @@ do_populate() {
                           includedeb ${DEBDISTRONAME} \
                           ${DEPLOY_DIR_DEB}/${p}_*.deb
         done
-
-        sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
-        for p in ${IMAGE_INSTALL}; do
-            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
-        done
-
-        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
-        sudo rm -rf ${IMAGE_ROOTFS}/deb
     fi
 }
 
-- 
2.1.4

Re: [PATCH v4 0/4] Basic binary cache implementation

[Claudius Heine]

Hi,

On 10/0	'2/2017 05:45 PM, Alexander Smirnov wrote:
> Hello everybody,
> 
> this series introduces basic binary caching for Isar.
> 
> There is a new layer: meta-isar-bin which is intended to be the binary
> cache. All the packages that are built in Isar are stored in this
> cache using reprepro.
> 
> Having the separate layer makes possible to manage this cache separately
> from the Isar build tree. So once you have built your packages, you could
> re-use this cache for further builds.

We still have to talk about your requirement for the cache to be 
sharable and your resulting implementation of it as a meta layer.

Since I still don't see the benefit of a sharable cache and think that 
putting binary packages into some kind of strange meta layer is not the 
right solution and could potentially create more problems that is solves 
because its very much in conflict with the fundamental idea of a meta layer.

Please give me a reason for this requirement.

And to be precise: Yes, a cache is useful and I fully support it, but 
having it sharable over multiple hosts is not and is potentially dangerous.

Claudius

> 
> Also this cache repo is now used to generate image filesystem via
> multistrap.
> 
> Changes since v3:
>   - Check return code from reprepro, continue loop only if EEXIST code is
>     returned. Othrwise build is aborted.
> 
> Changes since v2:
>   - Droped anonymous functions
>   - Added number of attempts to access apt cache before fail
> 
> Changes since v1:
>   - Total refactoring
>   - Split huge patches into small pieces
>   - Avoid races
> 
> With best regards,
> Alex
> 
> Alexander Smirnov (4):
>    meta-isar-bin: Add reprepro configs
>    meta-isar-bin: Generate cache repos
>    meta-isar-bin: Populate cache
>    meta-isar-bin: Install packages via multistrap
> 
>   meta-isar-bin/conf/layer.conf                      | 14 +++++
>   meta-isar-bin/files/distributions.in               |  3 ++
>   meta-isar/conf/bblayers.conf.sample                |  1 +
>   meta-isar/conf/local.conf.sample                   |  4 ++
>   .../recipes-core/images/files/multistrap.conf.in   |  9 +++-
>   meta-isar/recipes-core/images/isar-image-base.bb   |  5 +-
>   meta/classes/ext4-img.bbclass                      |  2 +-
>   meta/classes/image.bbclass                         | 62 +++++++++++++++++++---
>   8 files changed, 89 insertions(+), 11 deletions(-)
>   create mode 100644 meta-isar-bin/conf/layer.conf
>   create mode 100644 meta-isar-bin/files/distributions.in
> 

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 2/4] meta-isar-bin: Generate cache repos

[Henning Schild]

Am Mon, 2 Oct 2017 18:45:29 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Generate repos for each configuration in multiconfig.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar-bin/conf/layer.conf    |  3 +++
>  meta-isar/conf/local.conf.sample |  4 ++++
>  meta/classes/image.bbclass       | 35
> ++++++++++++++++++++++++++++++++++- 3 files changed, 41
> insertions(+), 1 deletion(-)
> 
> diff --git a/meta-isar-bin/conf/layer.conf
> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
> --- a/meta-isar-bin/conf/layer.conf
> +++ b/meta-isar-bin/conf/layer.conf
> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>  
>  # Path to the configuration files templates used by `reprepro`
>  DEBFILESDIR ?= "${LAYERDIR}/files"
> +
> +# Path to the databases used by `reprepro`
> +DEBDBDIR ?= "${LAYERDIR}/db"
> diff --git a/meta-isar/conf/local.conf.sample
> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
>  #
>  # Default parallel jobs for bitbake:
>  BB_NUMBER_THREADS = "4"
> +
> +#
> +# Number of attempts to try to get reprepro lock for access to apt
> cache +REPREPRO_LOCK_ATTEMPTS = "16"
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index d30b139..b07775a 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -1,5 +1,5 @@
>  # This software is a part of ISAR.
> -# Copyright (C) 2015-2016 ilbers GmbH
> +# Copyright (C) 2015-2017 ilbers GmbH
>  
>  KERNEL_IMAGE ?= ""
>  INITRD_IMAGE ?= ""
> @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
>  do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>  do_cache_config[stamp-extra-info] = "${DISTRO}"
>  
> +call_reprepro() {
> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> +        #  According to `sh` manual page, shell exit statuses fall
> between
> +        # 0-255. The EEXIST error code is (-17), so casting to
> usigned 8-bit

cant you use python to get that number and use the name "EEXIST" instead
of a hardcoded number? I think i sent a sniplet in an earlier mail.

> +        # integer results value (239).
> +        eexist="239"
> +        retval="0"
> +        reprepro --waitforlock 1 $* || retval="$?"

The waitforlock should be skipped altogether, which means try it once.
The current version means try twice. I got that wrong when i suggested
to go for "1", sorry.

> +        # If reprepro has failed to get database lock, it returns
> EEXIST code.
> +        # In this case we continue trying to get lock until max
> amount of
> +        # attempts is reached.
> +        if [ $retval -eq $eexist ]; then
> +            bbwarn "Failed to get reprepro lock, trying again..."

Are there other cases where reprepro returns -EEXIST?

Henning

> +        else
> +            break
> +        fi
> +    done
> +
> +    if [ $retval -ne 0 ]; then
> +        bbfatal "reprepro failed"
> +    fi
> +}
> +
>  # Generate reprepro config for current distro if it doesn't exist.
> Once it's # generated, this task should do nothing.
>  do_cache_config() {
> @@ -21,6 +45,15 @@ do_cache_config() {
>          sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>              ${DEBFILESDIR}/distributions.in >
> ${CACHE_CONF_DIR}/distributions fi
> +
> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
> +    path_databases="${DEBDBDIR}/${DISTRO}"
> +
> +    if [ ! -d "${path_databases}" ]; then
> +        call_reprepro -b ${path_cache} \
> +                      --dbdir ${path_databases} \
> +                      export ${DEBDISTRONAME}
> +    fi
>  }
>  
>  addtask cache_config before do_fetch

Re: [PATCH 4/4] meta-isar-bin: Install packages via multistrap

[Henning Schild]

Am Mon, 2 Oct 2017 18:45:31 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Install custom packages to target image filesystem using multistrap.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
>  meta-isar/recipes-core/images/isar-image-base.bb       |  5 ++++-
>  meta/classes/ext4-img.bbclass                          |  2 +-
>  meta/classes/image.bbclass                             | 10
> ---------- 4 files changed, 13 insertions(+), 13 deletions(-)
> 
> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
> b/meta-isar/recipes-core/images/files/multistrap.conf.in index
> 2d08c36..38d53cf 100644 ---
> a/meta-isar/recipes-core/images/files/multistrap.conf.in +++
> b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7
> @@ [General]
>  noauth=true
>  unpack=true
> -bootstrap=##DISTRO##
> +bootstrap=##DISTRO## Isar
>  aptsources=##DISTRO##
>  configscript=##CONFIG_SCRIPT##
>  setupscript=##SETUP_SCRIPT##
> @@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
>  components=##DISTRO_COMPONENTS##
>  packages=##IMAGE_PREINSTALL##
>  omitdebsrc=true
> +
> +[Isar]
> +packages=##IMAGE_INSTALL##
> +source=##DEPLOY_DIR_DEB##
> +suite=##ISAR_DISTRO_SUITE##
> +components=main
> +omitdebsrc=true
> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
> b/meta-isar/recipes-core/images/isar-image-base.bb index
> f4b0125..3b29a53 100644 ---
> a/meta-isar/recipes-core/images/isar-image-base.bb +++
> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -46,6 +46,9 @@
> do_rootfs() { -e
> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
> -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ -e
> 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
> +        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
> +        -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g'
> \
> +        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>             "${WORKDIR}/multistrap.conf.in" >
> "${WORKDIR}/multistrap.conf" 
>      # Create root filesystem
> @@ -57,4 +60,4 @@ do_rootfs() {
>      sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
>  }
>  
> -addtask rootfs before do_populate after do_unpack
> +addtask rootfs before do_build after do_populate
> diff --git a/meta/classes/ext4-img.bbclass
> b/meta/classes/ext4-img.bbclass index 3e3768b..9fa9aa5 100644
> --- a/meta/classes/ext4-img.bbclass
> +++ b/meta/classes/ext4-img.bbclass
> @@ -34,4 +34,4 @@ do_ext4_image() {
>      fi
>  }
>  
> -addtask ext4_image before do_build after do_populate
> +addtask ext4_image before do_build after do_rootfs
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index cf819e6..ff8a370 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -70,16 +70,6 @@ do_populate() {
>                            includedeb ${DEBDISTRONAME} \
>                            ${DEPLOY_DIR_DEB}/${p}_*.deb
>          done
> -
> -        sudo mkdir -p ${IMAGE_ROOTFS}/deb
> -
> -        for p in ${IMAGE_INSTALL}; do
> -            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
> -        done
> -
> -        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
> -
> -        sudo rm -rf ${IMAGE_ROOTFS}/deb

The comment of what do_populate does is wrong after that.

Henning

>      fi
>  }
>  

Re: [PATCH 2/4] meta-isar-bin: Generate cache repos

[Alexander Smirnov]

On 10/04/2017 12:03 PM, Henning Schild wrote:
> Am Mon, 2 Oct 2017 18:45:29 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Generate repos for each configuration in multiconfig.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar-bin/conf/layer.conf    |  3 +++
>>   meta-isar/conf/local.conf.sample |  4 ++++
>>   meta/classes/image.bbclass       | 35
>> ++++++++++++++++++++++++++++++++++- 3 files changed, 41
>> insertions(+), 1 deletion(-)
>>
>> diff --git a/meta-isar-bin/conf/layer.conf
>> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
>> --- a/meta-isar-bin/conf/layer.conf
>> +++ b/meta-isar-bin/conf/layer.conf
>> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
>>   
>>   # Path to the configuration files templates used by `reprepro`
>>   DEBFILESDIR ?= "${LAYERDIR}/files"
>> +
>> +# Path to the databases used by `reprepro`
>> +DEBDBDIR ?= "${LAYERDIR}/db"
>> diff --git a/meta-isar/conf/local.conf.sample
>> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
>> --- a/meta-isar/conf/local.conf.sample
>> +++ b/meta-isar/conf/local.conf.sample
>> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
>>   #
>>   # Default parallel jobs for bitbake:
>>   BB_NUMBER_THREADS = "4"
>> +
>> +#
>> +# Number of attempts to try to get reprepro lock for access to apt
>> cache +REPREPRO_LOCK_ATTEMPTS = "16"
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index d30b139..b07775a 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -1,5 +1,5 @@
>>   # This software is a part of ISAR.
>> -# Copyright (C) 2015-2016 ilbers GmbH
>> +# Copyright (C) 2015-2017 ilbers GmbH
>>   
>>   KERNEL_IMAGE ?= ""
>>   INITRD_IMAGE ?= ""
>> @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
>>   do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>>   do_cache_config[stamp-extra-info] = "${DISTRO}"
>>   
>> +call_reprepro() {
>> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
>> +        #  According to `sh` manual page, shell exit statuses fall
>> between
>> +        # 0-255. The EEXIST error code is (-17), so casting to
>> usigned 8-bit
> 
> cant you use python to get that number and use the name "EEXIST" instead
> of a hardcoded number? I think i sent a sniplet in an earlier mail.

No we can't. In python I've checked the code EEXIST, and it is (17), 
while return value from subprocess call is still (239). So using python 
here is only overhead in correct wrapping of reprepro call.

> 
>> +        # integer results value (239).
>> +        eexist="239"
>> +        retval="0"
>> +        reprepro --waitforlock 1 $* || retval="$?"
> 
> The waitforlock should be skipped altogether, which means try it once.
> The current version means try twice. I got that wrong when i suggested
> to go for "1", sorry.

'waitforlock 1' adds 10 seconds delay between attempts. So the idea to 
use this form to avoid extra sleep calls with magic timeout value.

> 
>> +        # If reprepro has failed to get database lock, it returns
>> EEXIST code.
>> +        # In this case we continue trying to get lock until max
>> amount of
>> +        # attempts is reached.
>> +        if [ $retval -eq $eexist ]; then
>> +            bbwarn "Failed to get reprepro lock, trying again..."
> 
> Are there other cases where reprepro returns -EEXIST?
> 

AFAIK there is only one case when reprepro could return EEXIST - it's 
morgue feature which is not used here. All the other EEXIST checks are 
processed internally.

Alex

> Henning
> 
>> +        else
>> +            break
>> +        fi
>> +    done
>> +
>> +    if [ $retval -ne 0 ]; then
>> +        bbfatal "reprepro failed"
>> +    fi
>> +}
>> +
>>   # Generate reprepro config for current distro if it doesn't exist.
>> Once it's # generated, this task should do nothing.
>>   do_cache_config() {
>> @@ -21,6 +45,15 @@ do_cache_config() {
>>           sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>>               ${DEBFILESDIR}/distributions.in >
>> ${CACHE_CONF_DIR}/distributions fi
>> +
>> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
>> +    path_databases="${DEBDBDIR}/${DISTRO}"
>> +
>> +    if [ ! -d "${path_databases}" ]; then
>> +        call_reprepro -b ${path_cache} \
>> +                      --dbdir ${path_databases} \
>> +                      export ${DEBDISTRONAME}
>> +    fi
>>   }
>>   
>>   addtask cache_config before do_fetch
> 

Re: [PATCH 4/4] meta-isar-bin: Install packages via multistrap

[Alexander Smirnov]

On 10/04/2017 12:05 PM, Henning Schild wrote:
> Am Mon, 2 Oct 2017 18:45:31 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Install custom packages to target image filesystem using multistrap.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
>>   meta-isar/recipes-core/images/isar-image-base.bb       |  5 ++++-
>>   meta/classes/ext4-img.bbclass                          |  2 +-
>>   meta/classes/image.bbclass                             | 10
>> ---------- 4 files changed, 13 insertions(+), 13 deletions(-)
>>
>> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in
>> b/meta-isar/recipes-core/images/files/multistrap.conf.in index
>> 2d08c36..38d53cf 100644 ---
>> a/meta-isar/recipes-core/images/files/multistrap.conf.in +++
>> b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7
>> @@ [General]
>>   noauth=true
>>   unpack=true
>> -bootstrap=##DISTRO##
>> +bootstrap=##DISTRO## Isar
>>   aptsources=##DISTRO##
>>   configscript=##CONFIG_SCRIPT##
>>   setupscript=##SETUP_SCRIPT##
>> @@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
>>   components=##DISTRO_COMPONENTS##
>>   packages=##IMAGE_PREINSTALL##
>>   omitdebsrc=true
>> +
>> +[Isar]
>> +packages=##IMAGE_INSTALL##
>> +source=##DEPLOY_DIR_DEB##
>> +suite=##ISAR_DISTRO_SUITE##
>> +components=main
>> +omitdebsrc=true
>> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb
>> b/meta-isar/recipes-core/images/isar-image-base.bb index
>> f4b0125..3b29a53 100644 ---
>> a/meta-isar/recipes-core/images/isar-image-base.bb +++
>> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -46,6 +46,9 @@
>> do_rootfs() { -e
>> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
>> -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ -e
>> 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
>> +        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
>> +        -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g'
>> \
>> +        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
>>              "${WORKDIR}/multistrap.conf.in" >
>> "${WORKDIR}/multistrap.conf"
>>       # Create root filesystem
>> @@ -57,4 +60,4 @@ do_rootfs() {
>>       sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
>>   }
>>   
>> -addtask rootfs before do_populate after do_unpack
>> +addtask rootfs before do_build after do_populate
>> diff --git a/meta/classes/ext4-img.bbclass
>> b/meta/classes/ext4-img.bbclass index 3e3768b..9fa9aa5 100644
>> --- a/meta/classes/ext4-img.bbclass
>> +++ b/meta/classes/ext4-img.bbclass
>> @@ -34,4 +34,4 @@ do_ext4_image() {
>>       fi
>>   }
>>   
>> -addtask ext4_image before do_build after do_populate
>> +addtask ext4_image before do_build after do_rootfs
>> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
>> index cf819e6..ff8a370 100644
>> --- a/meta/classes/image.bbclass
>> +++ b/meta/classes/image.bbclass
>> @@ -70,16 +70,6 @@ do_populate() {
>>                             includedeb ${DEBDISTRONAME} \
>>                             ${DEPLOY_DIR_DEB}/${p}_*.deb
>>           done
>> -
>> -        sudo mkdir -p ${IMAGE_ROOTFS}/deb
>> -
>> -        for p in ${IMAGE_INSTALL}; do
>> -            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
>> -        done
>> -
>> -        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
>> -
>> -        sudo rm -rf ${IMAGE_ROOTFS}/deb
> 
> The comment of what do_populate does is wrong after that.

Thanks for this hint.

> 
>>       fi
>>   }
>>   

Re: [PATCH v4 0/4] Basic binary cache implementation

[Jan Kiszka]

On 2017-10-04 10:32, [ext] Claudius Heine wrote:
> Hi,
> 
> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
>> Hello everybody,
>>
>> this series introduces basic binary caching for Isar.
>>
>> There is a new layer: meta-isar-bin which is intended to be the binary
>> cache. All the packages that are built in Isar are stored in this
>> cache using reprepro.
>>
>> Having the separate layer makes possible to manage this cache separately
>> from the Isar build tree. So once you have built your packages, you could
>> re-use this cache for further builds.
> 
> We still have to talk about your requirement for the cache to be
> sharable and your resulting implementation of it as a meta layer.
> 
> Since I still don't see the benefit of a sharable cache and think that
> putting binary packages into some kind of strange meta layer is not the
> right solution and could potentially create more problems that is solves
> because its very much in conflict with the fundamental idea of a meta
> layer.

Indeed, state like this (disclaimer: i didn't look into all details), it
sounds like a weird architecture to me as well. Thinking of an artifact
cache, I would rather expect something like OE's sstate that one may
preserve across builds, share between devs, or whatever. If it's there,
the build system consults it, if not, it rebuilds. But sstate is not a
layer, for some good reasons.

Jan
-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

Re: [PATCH 2/4] meta-isar-bin: Generate cache repos

[Henning Schild]

Am Wed, 4 Oct 2017 13:57:23 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 10/04/2017 12:03 PM, Henning Schild wrote:
> > Am Mon, 2 Oct 2017 18:45:29 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Generate repos for each configuration in multiconfig.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta-isar-bin/conf/layer.conf    |  3 +++
> >>   meta-isar/conf/local.conf.sample |  4 ++++
> >>   meta/classes/image.bbclass       | 35
> >> ++++++++++++++++++++++++++++++++++- 3 files changed, 41
> >> insertions(+), 1 deletion(-)
> >>
> >> diff --git a/meta-isar-bin/conf/layer.conf
> >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644
> >> --- a/meta-isar-bin/conf/layer.conf
> >> +++ b/meta-isar-bin/conf/layer.conf
> >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt"
> >>   
> >>   # Path to the configuration files templates used by `reprepro`
> >>   DEBFILESDIR ?= "${LAYERDIR}/files"
> >> +
> >> +# Path to the databases used by `reprepro`
> >> +DEBDBDIR ?= "${LAYERDIR}/db"
> >> diff --git a/meta-isar/conf/local.conf.sample
> >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
> >> --- a/meta-isar/conf/local.conf.sample
> >> +++ b/meta-isar/conf/local.conf.sample
> >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
> >>   #
> >>   # Default parallel jobs for bitbake:
> >>   BB_NUMBER_THREADS = "4"
> >> +
> >> +#
> >> +# Number of attempts to try to get reprepro lock for access to apt
> >> cache +REPREPRO_LOCK_ATTEMPTS = "16"
> >> diff --git a/meta/classes/image.bbclass
> >> b/meta/classes/image.bbclass index d30b139..b07775a 100644
> >> --- a/meta/classes/image.bbclass
> >> +++ b/meta/classes/image.bbclass
> >> @@ -1,5 +1,5 @@
> >>   # This software is a part of ISAR.
> >> -# Copyright (C) 2015-2016 ilbers GmbH
> >> +# Copyright (C) 2015-2017 ilbers GmbH
> >>   
> >>   KERNEL_IMAGE ?= ""
> >>   INITRD_IMAGE ?= ""
> >> @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf"
> >>   do_cache_config[dirs] = "${CACHE_CONF_DIR}"
> >>   do_cache_config[stamp-extra-info] = "${DISTRO}"
> >>   
> >> +call_reprepro() {
> >> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> >> +        #  According to `sh` manual page, shell exit statuses fall
> >> between
> >> +        # 0-255. The EEXIST error code is (-17), so casting to
> >> usigned 8-bit  
> > 
> > cant you use python to get that number and use the name "EEXIST"
> > instead of a hardcoded number? I think i sent a sniplet in an
> > earlier mail.  
> 
> No we can't. In python I've checked the code EEXIST, and it is (17), 
> while return value from subprocess call is still (239). So using
> python here is only overhead in correct wrapping of reprepro call.

Just do the math in python, skip the long comment and get rid of the
magic number.
python -c "import errno; print(256-errno.EEXIST)"

> >   
> >> +        # integer results value (239).
> >> +        eexist="239"
> >> +        retval="0"
> >> +        reprepro --waitforlock 1 $* || retval="$?"  
> > 
> > The waitforlock should be skipped altogether, which means try it
> > once. The current version means try twice. I got that wrong when i
> > suggested to go for "1", sorry.  
> 
> 'waitforlock 1' adds 10 seconds delay between attempts. So the idea
> to use this form to avoid extra sleep calls with magic timeout value.

Now you have two nested loops again and the name REPREPRO_LOCK_ATTEMPTS
is wrong, because you do it 2*REPREPRO_LOCK_ATTEMPTS. Either divide
that number by two or introduce a sleep on the error-path.
 
> >   
> >> +        # If reprepro has failed to get database lock, it returns
> >> EEXIST code.
> >> +        # In this case we continue trying to get lock until max
> >> amount of
> >> +        # attempts is reached.
> >> +        if [ $retval -eq $eexist ]; then
> >> +            bbwarn "Failed to get reprepro lock, trying
> >> again..."  
> > 
> > Are there other cases where reprepro returns -EEXIST?
> >   
> 
> AFAIK there is only one case when reprepro could return EEXIST - it's 
> morgue feature which is not used here. All the other EEXIST checks
> are processed internally.

Ok, just wanted to make sure someone checked that.

Henning

> Alex
> 
> > Henning
> >   
> >> +        else
> >> +            break
> >> +        fi
> >> +    done
> >> +
> >> +    if [ $retval -ne 0 ]; then
> >> +        bbfatal "reprepro failed"
> >> +    fi
> >> +}
> >> +
> >>   # Generate reprepro config for current distro if it doesn't
> >> exist. Once it's # generated, this task should do nothing.
> >>   do_cache_config() {
> >> @@ -21,6 +45,15 @@ do_cache_config() {
> >>           sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
> >>               ${DEBFILESDIR}/distributions.in >
> >> ${CACHE_CONF_DIR}/distributions fi
> >> +
> >> +    path_cache="${DEBCACHEDIR}/${DISTRO}"
> >> +    path_databases="${DEBDBDIR}/${DISTRO}"
> >> +
> >> +    if [ ! -d "${path_databases}" ]; then
> >> +        call_reprepro -b ${path_cache} \
> >> +                      --dbdir ${path_databases} \
> >> +                      export ${DEBDISTRONAME}
> >> +    fi
> >>   }
> >>   
> >>   addtask cache_config before do_fetch  
> >   

Re: [PATCH v4 0/4] Basic binary cache implementation

[Alexander Smirnov]

On 10/04/2017 02:57 PM, Jan Kiszka wrote:
> On 2017-10-04 10:32, [ext] Claudius Heine wrote:
>> Hi,
>>
>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
>>> Hello everybody,
>>>
>>> this series introduces basic binary caching for Isar.
>>>
>>> There is a new layer: meta-isar-bin which is intended to be the binary
>>> cache. All the packages that are built in Isar are stored in this
>>> cache using reprepro.
>>>
>>> Having the separate layer makes possible to manage this cache separately
>>> from the Isar build tree. So once you have built your packages, you could
>>> re-use this cache for further builds.
>>
>> We still have to talk about your requirement for the cache to be
>> sharable and your resulting implementation of it as a meta layer.
>>
>> Since I still don't see the benefit of a sharable cache and think that
>> putting binary packages into some kind of strange meta layer is not the
>> right solution and could potentially create more problems that is solves
>> because its very much in conflict with the fundamental idea of a meta
>> layer.

The last statement is a bit strange for me. The idea of meta layers is 
to split software stack into logical layers that could be 
enabled/disabled on demand. Having binaries or recipes doesn't matter 
here, it's just the way how software is provided to build.

> 
> Indeed, state like this (disclaimer: i didn't look into all details), it
> sounds like a weird architecture to me as well. Thinking of an artifact
> cache, I would rather expect something like OE's sstate that one may
> preserve across builds, share between devs, or whatever. If it's there,
> the build system consults it, if not, it rebuilds. But sstate is not a
> layer, for some good reasons.
> 

Let me again summarize major points here. Isar is the build system 
designed to work with binary packages. That's the key feature of this 
product and main difference from OE-like systems. The Isar architecture 
assumes to be designed around binary packages, what provides absolutely 
different approach how binary images could be generated using bitbake.

According to the Claudius's vision, that everything should always be 
built from sources - Isar is not the best option here, Yocto and OE are 
much better, because they were initially designed for this work. 
Attempts to apply OE design and philosophy to Isar could have negative 
impact and limit possible ways to implement custom features, because OE 
wasn't designed for this features.
Also I wonder why we are able to use binary Debian packages, while the 
rest of software should be built from sources.

This series introduces support for binary apt repositories in Isar. This 
feature is response to the following cases:

1. Publish custom apt layer in web. When I need to install some custom 
software to my Debian system, I frequently see instructions like:
  - Add 'blablabla' to your /etc/apt/source.list
  - Run apt-get update
  - Run apt-get install

That's for example how Jenkins and GitLab are distributed:

   https://pkg.jenkins.io/debian/

The management of this public apt could require some manual effort, 
while Isar (in plans) could update it automatically, so you update 
source code for your product, run Isar and it provides complete apt that 
you could upload to your server.

2. Implement easy software update. On our host systems we usually do not 
flash whole image if we need to update some packages. The update is 
performed using 'apt-get install' (or by similar tool). But in this case 
  binary repo is needed, what could be again provided by Isar.

Please do not limit Isar by deep embedded application, host systems also 
could be managed by Isar.

3. In long term, help Debian community to manage Debian infrastructure, 
what includes publishing apt updates.

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v4 0/4] Basic binary cache implementation

[Henning Schild]

Am Wed, 4 Oct 2017 17:29:08 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 10/04/2017 02:57 PM, Jan Kiszka wrote:
> > On 2017-10-04 10:32, [ext] Claudius Heine wrote:  
> >> Hi,
> >>
> >> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:  
> >>> Hello everybody,
> >>>
> >>> this series introduces basic binary caching for Isar.
> >>>
> >>> There is a new layer: meta-isar-bin which is intended to be the
> >>> binary cache. All the packages that are built in Isar are stored
> >>> in this cache using reprepro.
> >>>
> >>> Having the separate layer makes possible to manage this cache
> >>> separately from the Isar build tree. So once you have built your
> >>> packages, you could re-use this cache for further builds.  
> >>
> >> We still have to talk about your requirement for the cache to be
> >> sharable and your resulting implementation of it as a meta layer.
> >>
> >> Since I still don't see the benefit of a sharable cache and think
> >> that putting binary packages into some kind of strange meta layer
> >> is not the right solution and could potentially create more
> >> problems that is solves because its very much in conflict with the
> >> fundamental idea of a meta layer.  
> 
> The last statement is a bit strange for me. The idea of meta layers
> is to split software stack into logical layers that could be 
> enabled/disabled on demand. Having binaries or recipes doesn't matter 
> here, it's just the way how software is provided to build.

Binaries that can be rebuild should not become part of the build
system. The need to distribute them to the targets or other
developers/images later is what we all agree on. But that should be
mostly outside of isar.

Building an Image will result in the Image and a repo. Both have to
somehow make their way to the target, but that is somewhat out of scope
of the build system.

> > 
> > Indeed, state like this (disclaimer: i didn't look into all
> > details), it sounds like a weird architecture to me as well.
> > Thinking of an artifact cache, I would rather expect something like
> > OE's sstate that one may preserve across builds, share between
> > devs, or whatever. If it's there, the build system consults it, if
> > not, it rebuilds. But sstate is not a layer, for some good reasons.
> >   
> 
> Let me again summarize major points here. Isar is the build system 
> designed to work with binary packages. That's the key feature of this 
> product and main difference from OE-like systems. The Isar
> architecture assumes to be designed around binary packages, what
> provides absolutely different approach how binary images could be
> generated using bitbake.
> 
> According to the Claudius's vision, that everything should always be 
> built from sources - Isar is not the best option here, Yocto and OE
> are much better, because they were initially designed for this work. 
> Attempts to apply OE design and philosophy to Isar could have
> negative impact and limit possible ways to implement custom features,
> because OE wasn't designed for this features.
> Also I wonder why we are able to use binary Debian packages, while
> the rest of software should be built from sources.
> 
> This series introduces support for binary apt repositories in Isar.
> This feature is response to the following cases:
> 
> 1. Publish custom apt layer in web. When I need to install some
> custom software to my Debian system, I frequently see instructions
> like:
>   - Add 'blablabla' to your /etc/apt/source.list
>   - Run apt-get update
>   - Run apt-get install
> 
> That's for example how Jenkins and GitLab are distributed:
> 
>    https://pkg.jenkins.io/debian/

I think we all get that and agree on it.

> The management of this public apt could require some manual effort, 
> while Isar (in plans) could update it automatically, so you update 
> source code for your product, run Isar and it provides complete apt
> that you could upload to your server.

Exactly, those manual steps or custom scripts are outside the scope of
Isar. Just like the flashing of the Images onto the targets.
 
> 2. Implement easy software update. On our host systems we usually do
> not flash whole image if we need to update some packages. The update
> is performed using 'apt-get install' (or by similar tool). But in
> this case binary repo is needed, what could be again provided by Isar.
> 
> Please do not limit Isar by deep embedded application, host systems
> also could be managed by Isar.

If the repo ends up in the output directory all those scenarios are
still possible, _and_ your source tree will not contain build-output.
 
Henning

> 3. In long term, help Debian community to manage Debian
> infrastructure, what includes publishing apt updates.
> 

Re: [PATCH v4 0/4] Basic binary cache implementation

[Alexander Smirnov]

Hi,

On 10/04/2017 06:40 PM, Henning Schild wrote:
> Am Wed, 4 Oct 2017 17:29:08 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 10/04/2017 02:57 PM, Jan Kiszka wrote:
>>> On 2017-10-04 10:32, [ext] Claudius Heine wrote:
>>>> Hi,
>>>>
>>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
>>>>> Hello everybody,
>>>>>
>>>>> this series introduces basic binary caching for Isar.
>>>>>
>>>>> There is a new layer: meta-isar-bin which is intended to be the
>>>>> binary cache. All the packages that are built in Isar are stored
>>>>> in this cache using reprepro.
>>>>>
>>>>> Having the separate layer makes possible to manage this cache
>>>>> separately from the Isar build tree. So once you have built your
>>>>> packages, you could re-use this cache for further builds.
>>>>
>>>> We still have to talk about your requirement for the cache to be
>>>> sharable and your resulting implementation of it as a meta layer.
>>>>
>>>> Since I still don't see the benefit of a sharable cache and think
>>>> that putting binary packages into some kind of strange meta layer
>>>> is not the right solution and could potentially create more
>>>> problems that is solves because its very much in conflict with the
>>>> fundamental idea of a meta layer.
>>
>> The last statement is a bit strange for me. The idea of meta layers
>> is to split software stack into logical layers that could be
>> enabled/disabled on demand. Having binaries or recipes doesn't matter
>> here, it's just the way how software is provided to build.
> 
> Binaries that can be rebuild should not become part of the build
> system. The need to distribute them to the targets or other
> developers/images later is what we all agree on. But that should be
> mostly outside of isar.

Each binary could be re-built from sources (at least for Debian :-) ). 
Moreover Debian provides sources for them, so they are open for use.

But for me it looks like:

  - Installing binaries to target from 'ftp.debian.org' is allowed.
  - Installing binary dependencies for custom built package from 
'ftp.debian.org' is allowed.
  - Installing my binary application from 'my.debian.repo' is not 
allowed. I have to build it from sources.

What is the idea here? For me it's something like traditional OE 
philosophy, but without benefit for Isar.

One of the typical usecase: if I'm, for example, developing 'hello' 
application, why it's so critical for me to build, lets say, kernel from 
sources?

Alex

> 
> Building an Image will result in the Image and a repo. Both have to
> somehow make their way to the target, but that is somewhat out of scope
> of the build system.
> 
>>>
>>> Indeed, state like this (disclaimer: i didn't look into all
>>> details), it sounds like a weird architecture to me as well.
>>> Thinking of an artifact cache, I would rather expect something like
>>> OE's sstate that one may preserve across builds, share between
>>> devs, or whatever. If it's there, the build system consults it, if
>>> not, it rebuilds. But sstate is not a layer, for some good reasons.
>>>    
>>
>> Let me again summarize major points here. Isar is the build system
>> designed to work with binary packages. That's the key feature of this
>> product and main difference from OE-like systems. The Isar
>> architecture assumes to be designed around binary packages, what
>> provides absolutely different approach how binary images could be
>> generated using bitbake.
>>
>> According to the Claudius's vision, that everything should always be
>> built from sources - Isar is not the best option here, Yocto and OE
>> are much better, because they were initially designed for this work.
>> Attempts to apply OE design and philosophy to Isar could have
>> negative impact and limit possible ways to implement custom features,
>> because OE wasn't designed for this features.
>> Also I wonder why we are able to use binary Debian packages, while
>> the rest of software should be built from sources.
>>
>> This series introduces support for binary apt repositories in Isar.
>> This feature is response to the following cases:
>>
>> 1. Publish custom apt layer in web. When I need to install some
>> custom software to my Debian system, I frequently see instructions
>> like:
>>    - Add 'blablabla' to your /etc/apt/source.list
>>    - Run apt-get update
>>    - Run apt-get install
>>
>> That's for example how Jenkins and GitLab are distributed:
>>
>>     https://pkg.jenkins.io/debian/
> 
> I think we all get that and agree on it.
> 
>> The management of this public apt could require some manual effort,
>> while Isar (in plans) could update it automatically, so you update
>> source code for your product, run Isar and it provides complete apt
>> that you could upload to your server.
> 
> Exactly, those manual steps or custom scripts are outside the scope of
> Isar. Just like the flashing of the Images onto the targets.
>   
>> 2. Implement easy software update. On our host systems we usually do
>> not flash whole image if we need to update some packages. The update
>> is performed using 'apt-get install' (or by similar tool). But in
>> this case binary repo is needed, what could be again provided by Isar.
>>
>> Please do not limit Isar by deep embedded application, host systems
>> also could be managed by Isar.
> 
> If the repo ends up in the output directory all those scenarios are
> still possible, _and_ your source tree will not contain build-output.
>   
> Henning
> 
>> 3. In long term, help Debian community to manage Debian
>> infrastructure, what includes publishing apt updates.
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH v4 0/4] Basic binary cache implementation

[Henning Schild]

Am Wed, 4 Oct 2017 19:50:09 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi,
> 
> On 10/04/2017 06:40 PM, Henning Schild wrote:
> > Am Wed, 4 Oct 2017 17:29:08 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> On 10/04/2017 02:57 PM, Jan Kiszka wrote:  
> >>> On 2017-10-04 10:32, [ext] Claudius Heine wrote:  
> >>>> Hi,
> >>>>
> >>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:  
> >>>>> Hello everybody,
> >>>>>
> >>>>> this series introduces basic binary caching for Isar.
> >>>>>
> >>>>> There is a new layer: meta-isar-bin which is intended to be the
> >>>>> binary cache. All the packages that are built in Isar are stored
> >>>>> in this cache using reprepro.
> >>>>>
> >>>>> Having the separate layer makes possible to manage this cache
> >>>>> separately from the Isar build tree. So once you have built your
> >>>>> packages, you could re-use this cache for further builds.  
> >>>>
> >>>> We still have to talk about your requirement for the cache to be
> >>>> sharable and your resulting implementation of it as a meta layer.
> >>>>
> >>>> Since I still don't see the benefit of a sharable cache and think
> >>>> that putting binary packages into some kind of strange meta layer
> >>>> is not the right solution and could potentially create more
> >>>> problems that is solves because its very much in conflict with
> >>>> the fundamental idea of a meta layer.  
> >>
> >> The last statement is a bit strange for me. The idea of meta layers
> >> is to split software stack into logical layers that could be
> >> enabled/disabled on demand. Having binaries or recipes doesn't
> >> matter here, it's just the way how software is provided to build.  
> > 
> > Binaries that can be rebuild should not become part of the build
> > system. The need to distribute them to the targets or other
> > developers/images later is what we all agree on. But that should be
> > mostly outside of isar.  
> 
> Each binary could be re-built from sources (at least for Debian :-)
> ). Moreover Debian provides sources for them, so they are open for
> use.
> 
> But for me it looks like:
> 
>   - Installing binaries to target from 'ftp.debian.org' is allowed.
>   - Installing binary dependencies for custom built package from 
> 'ftp.debian.org' is allowed.
>   - Installing my binary application from 'my.debian.repo' is not 
> allowed. I have to build it from sources.

That is not only allowed it is absolutely wanted! But for that step you
would add that repo to your multistrap config or /etc/apt/, which is in
the scope of isar.
Building the repo is also in the scope of isar, but holding the
artefacts and publishing the repo outside the build-dir is not.

Henning

> What is the idea here? For me it's something like traditional OE 
> philosophy, but without benefit for Isar.
> 
> One of the typical usecase: if I'm, for example, developing 'hello' 
> application, why it's so critical for me to build, lets say, kernel
> from sources?
> 
> Alex
> 
> > 
> > Building an Image will result in the Image and a repo. Both have to
> > somehow make their way to the target, but that is somewhat out of
> > scope of the build system.
> >   
> >>>
> >>> Indeed, state like this (disclaimer: i didn't look into all
> >>> details), it sounds like a weird architecture to me as well.
> >>> Thinking of an artifact cache, I would rather expect something
> >>> like OE's sstate that one may preserve across builds, share
> >>> between devs, or whatever. If it's there, the build system
> >>> consults it, if not, it rebuilds. But sstate is not a layer, for
> >>> some good reasons. 
> >>
> >> Let me again summarize major points here. Isar is the build system
> >> designed to work with binary packages. That's the key feature of
> >> this product and main difference from OE-like systems. The Isar
> >> architecture assumes to be designed around binary packages, what
> >> provides absolutely different approach how binary images could be
> >> generated using bitbake.
> >>
> >> According to the Claudius's vision, that everything should always
> >> be built from sources - Isar is not the best option here, Yocto
> >> and OE are much better, because they were initially designed for
> >> this work. Attempts to apply OE design and philosophy to Isar
> >> could have negative impact and limit possible ways to implement
> >> custom features, because OE wasn't designed for this features.
> >> Also I wonder why we are able to use binary Debian packages, while
> >> the rest of software should be built from sources.
> >>
> >> This series introduces support for binary apt repositories in Isar.
> >> This feature is response to the following cases:
> >>
> >> 1. Publish custom apt layer in web. When I need to install some
> >> custom software to my Debian system, I frequently see instructions
> >> like:
> >>    - Add 'blablabla' to your /etc/apt/source.list
> >>    - Run apt-get update
> >>    - Run apt-get install
> >>
> >> That's for example how Jenkins and GitLab are distributed:
> >>
> >>     https://pkg.jenkins.io/debian/  
> > 
> > I think we all get that and agree on it.
> >   
> >> The management of this public apt could require some manual effort,
> >> while Isar (in plans) could update it automatically, so you update
> >> source code for your product, run Isar and it provides complete apt
> >> that you could upload to your server.  
> > 
> > Exactly, those manual steps or custom scripts are outside the scope
> > of Isar. Just like the flashing of the Images onto the targets.
> >     
> >> 2. Implement easy software update. On our host systems we usually
> >> do not flash whole image if we need to update some packages. The
> >> update is performed using 'apt-get install' (or by similar tool).
> >> But in this case binary repo is needed, what could be again
> >> provided by Isar.
> >>
> >> Please do not limit Isar by deep embedded application, host systems
> >> also could be managed by Isar.  
> > 
> > If the repo ends up in the output directory all those scenarios are
> > still possible, _and_ your source tree will not contain
> > build-output. 
> > Henning
> >   
> >> 3. In long term, help Debian community to manage Debian
> >> infrastructure, what includes publishing apt updates.
> >>  
> >   
> 

Re: [PATCH v4 0/4] Basic binary cache implementation

[Claudius Heine]

[-- Attachment #1: Type: text/plain, Size: 4820 bytes --]

Hi Alex,

On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote:
> On 10/04/2017 02:57 PM, Jan Kiszka wrote:
> > On 2017-10-04 10:32, [ext] Claudius Heine wrote:
> > > Hi,
> > > 
> > > On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
> > > > Hello everybody,
> > > > 
> > > > this series introduces basic binary caching for Isar.
> > > > 
> > > > There is a new layer: meta-isar-bin which is intended to be the
> > > > binary
> > > > cache. All the packages that are built in Isar are stored in
> > > > this
> > > > cache using reprepro.
> > > > 
> > > > Having the separate layer makes possible to manage this cache
> > > > separately
> > > > from the Isar build tree. So once you have built your packages,
> > > > you could
> > > > re-use this cache for further builds.
> > > 
> > > We still have to talk about your requirement for the cache to be
> > > sharable and your resulting implementation of it as a meta layer.
> > > 
> > > Since I still don't see the benefit of a sharable cache and think
> > > that
> > > putting binary packages into some kind of strange meta layer is
> > > not the
> > > right solution and could potentially create more problems that is
> > > solves
> > > because its very much in conflict with the fundamental idea of a
> > > meta
> > > layer.
> 
> The last statement is a bit strange for me. The idea of meta layers
> is 
> to split software stack into logical layers that could be 
> enabled/disabled on demand. Having binaries or recipes doesn't
> matter 
> here, it's just the way how software is provided to build.
> 
> > 
> > Indeed, state like this (disclaimer: i didn't look into all
> > details), it
> > sounds like a weird architecture to me as well. Thinking of an
> > artifact
> > cache, I would rather expect something like OE's sstate that one
> > may
> > preserve across builds, share between devs, or whatever. If it's
> > there,
> > the build system consults it, if not, it rebuilds. But sstate is
> > not a
> > layer, for some good reasons.
> > 
> 
> Let me again summarize major points here. Isar is the build system 
> designed to work with binary packages. That's the key feature of
> this 
> product and main difference from OE-like systems. The Isar
> architecture 
> assumes to be designed around binary packages, what provides
> absolutely 
> different approach how binary images could be generated using
> bitbake.
> 
> According to the Claudius's vision, that everything should always be 
> built from sources - Isar is not the best option here, Yocto and OE
> are 
> much better, because they were initially designed for this work. 
> Attempts to apply OE design and philosophy to Isar could have
> negative 
> impact and limit possible ways to implement custom features, because
> OE 
> wasn't designed for this features.
> Also I wonder why we are able to use binary Debian packages, while
> the 
> rest of software should be built from sources.

You are still not understanding me. I never said you have to build
every package from source code in isar. Instead I mean that you have to
build the root filesystem / images / packages ... ( every output of
isar) from all the available input of isar (upstream debian
repositories, meta-layers, source code repositories, etc.).

If you create intermediate products in the form of caches, then that is
ok. But if you put those intermediate products into repositories and
begin to distribute them, then this causes problems as I described
before.

Distributing intermediate products is always a bad idea and systems
should not be designed with this in mind or even advertise it. That has
nothing to do with 'OE' or 'Debian' or any other System, its just
general software developing and maintenance rules.

It is ok to to this occasionally on a case by case basis, but it should
not be a requirement or even an advertised feature.

> ...

In the rest of your email or this thread I still don't see any reason
why sharing caches is necessary.

Building the output from the input is fine once to fill the caches.
Just to be sure that you are in fact able to build the output from the
input. And I don't get why this is bad in your opinion.

I get that we can trust that Debian can build their packages from their
sources, so we don't need to test that, but at least test if we can
still build our own packages in isar.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

            PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153
                              Keyserver: hkp://pool.sks-keyservers.net

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: [PATCH v4 0/4] Basic binary cache implementation

[Alexander Smirnov]

On 10/04/2017 08:34 PM, Claudius Heine wrote:
> Hi Alex,
> 
> On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote:
>> On 10/04/2017 02:57 PM, Jan Kiszka wrote:
>>> On 2017-10-04 10:32, [ext] Claudius Heine wrote:
>>>> Hi,
>>>>
>>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
>>>>> Hello everybody,
>>>>>
>>>>> this series introduces basic binary caching for Isar.
>>>>>
>>>>> There is a new layer: meta-isar-bin which is intended to be the
>>>>> binary
>>>>> cache. All the packages that are built in Isar are stored in
>>>>> this
>>>>> cache using reprepro.
>>>>>
>>>>> Having the separate layer makes possible to manage this cache
>>>>> separately
>>>>> from the Isar build tree. So once you have built your packages,
>>>>> you could
>>>>> re-use this cache for further builds.
>>>>
>>>> We still have to talk about your requirement for the cache to be
>>>> sharable and your resulting implementation of it as a meta layer.
>>>>
>>>> Since I still don't see the benefit of a sharable cache and think
>>>> that
>>>> putting binary packages into some kind of strange meta layer is
>>>> not the
>>>> right solution and could potentially create more problems that is
>>>> solves
>>>> because its very much in conflict with the fundamental idea of a
>>>> meta
>>>> layer.
>>
>> The last statement is a bit strange for me. The idea of meta layers
>> is
>> to split software stack into logical layers that could be
>> enabled/disabled on demand. Having binaries or recipes doesn't
>> matter
>> here, it's just the way how software is provided to build.
>>
>>>
>>> Indeed, state like this (disclaimer: i didn't look into all
>>> details), it
>>> sounds like a weird architecture to me as well. Thinking of an
>>> artifact
>>> cache, I would rather expect something like OE's sstate that one
>>> may
>>> preserve across builds, share between devs, or whatever. If it's
>>> there,
>>> the build system consults it, if not, it rebuilds. But sstate is
>>> not a
>>> layer, for some good reasons.
>>>
>>
>> Let me again summarize major points here. Isar is the build system
>> designed to work with binary packages. That's the key feature of
>> this
>> product and main difference from OE-like systems. The Isar
>> architecture
>> assumes to be designed around binary packages, what provides
>> absolutely
>> different approach how binary images could be generated using
>> bitbake.
>>
>> According to the Claudius's vision, that everything should always be
>> built from sources - Isar is not the best option here, Yocto and OE
>> are
>> much better, because they were initially designed for this work.
>> Attempts to apply OE design and philosophy to Isar could have
>> negative
>> impact and limit possible ways to implement custom features, because
>> OE
>> wasn't designed for this features.
>> Also I wonder why we are able to use binary Debian packages, while
>> the
>> rest of software should be built from sources.
> 
> You are still not understanding me. I never said you have to build
> every package from source code in isar. Instead I mean that you have to
> build the root filesystem / images / packages ... ( every output of
> isar) from all the available input of isar (upstream debian
> repositories, meta-layers, source code repositories, etc.).

Yes, and I want to add my custom apt repo to the Isar, even if I have 
source code for packages in it.

> 
> If you create intermediate products in the form of caches, then that is
> ok. But if you put those intermediate products into repositories and
> begin to distribute them, then this causes problems as I described
> before.
> 

If you mean version-control repository, then nobody asks you to do this. 
If you mean apt repository, Isar should be able to export it with your 
product (application layer), like I've mentioned with examples in my mail.

> Distributing intermediate products is always a bad idea and systems
> should not be designed with this in mind or even advertise it. That has
> nothing to do with 'OE' or 'Debian' or any other System, its just
> general software developing and maintenance rules.

That's contradictory sentences for me. Debian distributes both:

  - Complete iso, USB etc... images
  - Apt repository

And both are intermediate in terms of your custom product.

So if you want to develop package for Debian, you do not need to build 
whole Debian from the scratch, you likely use intermediate product 
instead - install base system and tools from binaries.

Isar is based on intermediate product, i.e. Debian apt repository.

So I still don't understand, why you consider that 'apt-get install' 
from Debian repository is OK, but 'apt-get install' from custom 
repository is a bad idea?

> 
> It is ok to to this occasionally on a case by case basis, but it should
> not be a requirement or even an advertised feature.
> 
>> ...
> 
> In the rest of your email or this thread I still don't see any reason
> why sharing caches is necessary.
> 
> Building the output from the input is fine once to fill the caches.
> Just to be sure that you are in fact able to build the output from the
> input. And I don't get why this is bad in your opinion.
> 

Because I don't want to waste the time for the work that has been 
already done and tested.

Just for sure, the release builds should be run from the scratch. And 
integration testing should be performed.

Building everything from the scratch only guarantees that your package 
could be built, nothing more. But I'd prefer to delegate this task to CI.

> I get that we can trust that Debian can build their packages from their
> sources, so we don't need to test that, but at least test if we can
> still build our own packages in isar.

Sorry, didn't get this.

Alex

Re: [PATCH v4 0/4] Basic binary cache implementation

[Claudius Heine]

Hi Alex,

On 10/04/2017 08:47 PM, Alexander Smirnov wrote:
> On 10/04/2017 08:34 PM, Claudius Heine wrote:
>> Hi Alex,
>>
>> On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote:
>>> On 10/04/2017 02:57 PM, Jan Kiszka wrote:
>>>> On 2017-10-04 10:32, [ext] Claudius Heine wrote:
>>>>> Hi,
>>>>>
>>>>> On 10/0    '2/2017 05:45 PM, Alexander Smirnov wrote:
>>>>>> Hello everybody,
>>>>>>
>>>>>> this series introduces basic binary caching for Isar.
>>>>>>
>>>>>> There is a new layer: meta-isar-bin which is intended to be the
>>>>>> binary
>>>>>> cache. All the packages that are built in Isar are stored in
>>>>>> this
>>>>>> cache using reprepro.
>>>>>>
>>>>>> Having the separate layer makes possible to manage this cache
>>>>>> separately
>>>>>> from the Isar build tree. So once you have built your packages,
>>>>>> you could
>>>>>> re-use this cache for further builds.
>>>>>
>>>>> We still have to talk about your requirement for the cache to be
>>>>> sharable and your resulting implementation of it as a meta layer.
>>>>>
>>>>> Since I still don't see the benefit of a sharable cache and think
>>>>> that
>>>>> putting binary packages into some kind of strange meta layer is
>>>>> not the
>>>>> right solution and could potentially create more problems that is
>>>>> solves
>>>>> because its very much in conflict with the fundamental idea of a
>>>>> meta
>>>>> layer.
>>>
>>> The last statement is a bit strange for me. The idea of meta layers
>>> is
>>> to split software stack into logical layers that could be
>>> enabled/disabled on demand. Having binaries or recipes doesn't
>>> matter
>>> here, it's just the way how software is provided to build.
>>>
>>>>
>>>> Indeed, state like this (disclaimer: i didn't look into all
>>>> details), it
>>>> sounds like a weird architecture to me as well. Thinking of an
>>>> artifact
>>>> cache, I would rather expect something like OE's sstate that one
>>>> may
>>>> preserve across builds, share between devs, or whatever. If it's
>>>> there,
>>>> the build system consults it, if not, it rebuilds. But sstate is
>>>> not a
>>>> layer, for some good reasons.
>>>>
>>>
>>> Let me again summarize major points here. Isar is the build system
>>> designed to work with binary packages. That's the key feature of
>>> this
>>> product and main difference from OE-like systems. The Isar
>>> architecture
>>> assumes to be designed around binary packages, what provides
>>> absolutely
>>> different approach how binary images could be generated using
>>> bitbake.
>>>
>>> According to the Claudius's vision, that everything should always be
>>> built from sources - Isar is not the best option here, Yocto and OE
>>> are
>>> much better, because they were initially designed for this work.
>>> Attempts to apply OE design and philosophy to Isar could have
>>> negative
>>> impact and limit possible ways to implement custom features, because
>>> OE
>>> wasn't designed for this features.
>>> Also I wonder why we are able to use binary Debian packages, while
>>> the
>>> rest of software should be built from sources.
>>
>> You are still not understanding me. I never said you have to build
>> every package from source code in isar. Instead I mean that you have to
>> build the root filesystem / images / packages ... ( every output of
>> isar) from all the available input of isar (upstream debian
>> repositories, meta-layers, source code repositories, etc.).
> 
> Yes, and I want to add my custom apt repo to the Isar, even if I have 
> source code for packages in it.
> 
>>
>> If you create intermediate products in the form of caches, then that is
>> ok. But if you put those intermediate products into repositories and
>> begin to distribute them, then this causes problems as I described
>> before.
>>
> 
> If you mean version-control repository, then nobody asks you to do this. 
> If you mean apt repository, Isar should be able to export it with your 
> product (application layer), like I've mentioned with examples in my mail.
> 
>> Distributing intermediate products is always a bad idea and systems
>> should not be designed with this in mind or even advertise it. That has
>> nothing to do with 'OE' or 'Debian' or any other System, its just
>> general software developing and maintenance rules.
> 
> That's contradictory sentences for me. Debian distributes both:
> 
>   - Complete iso, USB etc... images
>   - Apt repository
> 
> And both are intermediate in terms of your custom product.

But thats a different feature. Yes eventually isar should export a 
tarball of the repository to the deploy dir or similar. But that is not 
what this patchset is about.

> 
> So if you want to develop package for Debian, you do not need to build 
> whole Debian from the scratch, you likely use intermediate product 
> instead - install base system and tools from binaries.
> 
> Isar is based on intermediate product, i.e. Debian apt repository.

The Debian repository is not a intermediate product for isar! But the 
resulting apt repository that is created by isar is a intermediate or 
final product of isar because it is generated by isar, therefor it 
should not be used as the input of a build process! But I am ok with it 
being used as a local cache! The difference is that the cache is not 
normally distributed to the user while the input is normally distributed 
to the user. Cache is internal to the build process while output and 
input is external.

> 
> So I still don't understand, why you consider that 'apt-get install' 
> from Debian repository is OK, but 'apt-get install' from custom 
> repository is a bad idea?

This discussion is frankly moving from frustrating to slightly infuriating.

Implementing the possibility to use additional apt repositories that are 
added to the multistrap.conf and are defined in a bitbake variable is 
also a totally fine feature. But that is also not what this patchset is 
about.

I don't appreciate you putting words in my mouth and using straw men 
arguments and moving the goalpost is not how we can discuss these things 
constructively.

> 
>>
>> It is ok to to this occasionally on a case by case basis, but it should
>> not be a requirement or even an advertised feature.
>>
>>> ...
>>
>> In the rest of your email or this thread I still don't see any reason
>> why sharing caches is necessary.
>>
>> Building the output from the input is fine once to fill the caches.
>> Just to be sure that you are in fact able to build the output from the
>> input. And I don't get why this is bad in your opinion.
>>
> 
> Because I don't want to waste the time for the work that has been 
> already done and tested.
> 
> Just for sure, the release builds should be run from the scratch. And 
> integration testing should be performed.
> 
> Building everything from the scratch only guarantees that your package 
> could be built, nothing more. But I'd prefer to delegate this task to CI.
> 
>> I get that we can trust that Debian can build their packages from their
>> sources, so we don't need to test that, but at least test if we can
>> still build our own packages in isar.
> 
> Sorry, didn't get this.

Yes I see, and I don't know how I can explain it any simpler.

Here is the main bad idea of this patch:

   - Using a meta-layer as a place to dump isar generated output at. 
meta-layers should only contain input data.

All my other arguments, that you don't seem to understand, are trying to 
explain why this is a bad idea.

Also I would really like to have a package cache, the ability to export 
the generated repository and the ability to used additional 
repositories. But the way this is integrated mixes input and output of 
the build system and makes many things unclear. So those 3 features 
should be handled separately.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de