[PATCH 0/4 v5] Isar apt deployment

[PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

Hi all,

this series switch Isar internal binary package processing to apt repository.
It performs this following:

1. Create repositories 'tmp/deploy/apt' for all architectures requested by
   multiconfig.

2. Generate reprepro database.

3. Put all the newly built packages to apt repository.

4. Pass this Isar repository to image multistrap.

Documentation will be updated after agreement on this implementation.

Changes since v4:
 - Drop meta-isar-bin layer.

With best regards,
Alex

Alexander Smirnov (4):
  apt: Generate configs for apt
  apt: Generate Isar reprepro database
  apt: Populate Isar apt
  apt: Install packages via multistrap

 meta-isar/conf/layer.conf                          | 13 ++++-
 meta-isar/conf/local.conf.sample                   |  4 ++
 .../recipes-core/images/files/distributions.in     |  3 +
 .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
 meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
 meta/classes/ext4-img.bbclass                      |  2 +-
 meta/classes/image.bbclass                         | 65 +++++++++++++++++++---
 7 files changed, 88 insertions(+), 14 deletions(-)
 create mode 100644 meta-isar/recipes-core/images/files/distributions.in

-- 
2.1.4

[PATCH 1/4] apt: Generate configs for apt

[Alexander Smirnov]

Generate reprepro configuration files for apt deployment.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/conf/layer.conf                            | 10 ++++++++--
 meta-isar/recipes-core/images/files/distributions.in |  3 +++
 meta/classes/image.bbclass                           | 17 ++++++++++++++++-
 3 files changed, 27 insertions(+), 3 deletions(-)
 create mode 100644 meta-isar/recipes-core/images/files/distributions.in

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index 72a103c..0950a34 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -1,5 +1,5 @@
 # This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
+# Copyright (C) 2015-2017 ilbers GmbH
 
 # We have a conf and classes directory, add to BBPATH
 BBPATH .= ":${LAYERDIR}"
@@ -13,6 +13,12 @@ BBFILE_PRIORITY_isar = "5"
 
 # This should only be incremented on significant changes that will
 # cause compatibility issues with other layers
-LAYERVERSION_isar = "2"
+LAYERVERSION_isar = "3"
 
 LAYERDIR_isar = "${LAYERDIR}"
+
+# Codename of the repository created by the caching class
+DEBDISTRONAME = "isar"
+
+# Path to the Isar apt repository
+DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
diff --git a/meta-isar/recipes-core/images/files/distributions.in b/meta-isar/recipes-core/images/files/distributions.in
new file mode 100644
index 0000000..cd214c6
--- /dev/null
+++ b/meta-isar/recipes-core/images/files/distributions.in
@@ -0,0 +1,3 @@
+Codename: {DISTRO_NAME}
+Architectures: i386 armhf amd64 source
+Components: main
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 5bf9524..57705a7 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -1,5 +1,5 @@
 # This software is a part of ISAR.
-# Copyright (C) 2015-2016 ilbers GmbH
+# Copyright (C) 2015-2017 ilbers GmbH
 
 KERNEL_IMAGE ?= ""
 INITRD_IMAGE ?= ""
@@ -10,6 +10,21 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
 
 inherit ${IMAGE_TYPE}
 
+CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
+do_cache_config[dirs] = "${CACHE_CONF_DIR}"
+do_cache_config[stamp-extra-info] = "${DISTRO}"
+
+# Generate reprepro config for current distro if it doesn't exist. Once it's
+# generated, this task should do nothing.
+do_cache_config() {
+    if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then
+        sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
+            ${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
+    fi
+}
+
+addtask cache_config before do_fetch
+
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 
 # Install Debian packages, that were built from sources
-- 
2.1.4

[PATCH 2/4] apt: Generate Isar reprepro database

[Alexander Smirnov]

Generate reprepro database for Isar apt repository.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/conf/layer.conf        |  3 +++
 meta-isar/conf/local.conf.sample |  4 ++++
 meta/classes/image.bbclass       | 34 ++++++++++++++++++++++++++++++++++
 3 files changed, 41 insertions(+)

diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
index 0950a34..cd42f06 100644
--- a/meta-isar/conf/layer.conf
+++ b/meta-isar/conf/layer.conf
@@ -22,3 +22,6 @@ DEBDISTRONAME = "isar"
 
 # Path to the Isar apt repository
 DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
+
+# Path to the Isar databases used by `reprepro`
+DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index a456b1b..660958f 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
 #
 # Default parallel jobs for bitbake:
 BB_NUMBER_THREADS = "4"
+
+#
+# Number of attempts to try to get reprepro lock for access to apt cache
+REPREPRO_LOCK_ATTEMPTS = "16"
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 57705a7..47f53a3 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
 
 inherit ${IMAGE_TYPE}
 
+call_reprepro() {
+    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
+        #  According to `sh` manual page, shell exit statuses fall between
+        # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
+        # integer results value (239).
+        eexist=$(python -c 'import errno; print(256-errno.EEXIST)')
+        retval="0"
+        reprepro $* || retval="$?"
+
+        # If reprepro has failed to get database lock, it returns EEXIST code.
+        # In this case we continue trying to get lock until max amount of
+        # attempts is reached.
+        if [ $retval -eq $eexist ]; then
+            bbwarn "Failed to get reprepro lock, trying again..."
+            sleep 5
+        else
+            break
+        fi
+    done
+
+    if [ $retval -ne 0 ]; then
+        bbfatal "reprepro failed"
+    fi
+}
+
 CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
 do_cache_config[dirs] = "${CACHE_CONF_DIR}"
 do_cache_config[stamp-extra-info] = "${DISTRO}"
@@ -21,6 +46,15 @@ do_cache_config() {
         sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
             ${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
     fi
+
+    path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
+    path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
+
+    if [ ! -d "${path_databases}" ]; then
+        call_reprepro -b ${path_cache} \
+                      --dbdir ${path_databases} \
+                      export ${DEBDISTRONAME}
+    fi
 }
 
 addtask cache_config before do_fetch
-- 
2.1.4

[PATCH 3/4] apt: Populate Isar apt

[Alexander Smirnov]

Populate newly built binary packages to Isar apt repository.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/image.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 47f53a3..610dd74 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -64,6 +64,13 @@ do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 # Install Debian packages, that were built from sources
 do_populate() {
     if [ -n "${IMAGE_INSTALL}" ]; then
+        for p in ${IMAGE_INSTALL}; do
+            call_reprepro -b ${DEPLOY_DIR_APT}/${DISTRO} \
+                          --dbdir ${DEPLOY_DIR_DB}/${DISTRO} \
+                          -C main \
+                          includedeb ${DEBDISTRONAME} \
+                          ${DEPLOY_DIR_DEB}/${p}_*.deb
+        done
         sudo mkdir -p ${IMAGE_ROOTFS}/deb
 
         for p in ${IMAGE_INSTALL}; do
-- 
2.1.4

[PATCH 4/4] apt: Install packages via multistrap

[Alexander Smirnov]

Install custom packages to target image filesystem using multistrap.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
 meta-isar/recipes-core/images/isar-image-base.bb       |  6 +++++-
 meta/classes/ext4-img.bbclass                          |  2 +-
 meta/classes/image.bbclass                             | 11 +----------
 4 files changed, 15 insertions(+), 13 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..491e5a2 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
 [General]
 noauth=true
 unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
 aptsources=##DISTRO##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
 components=##DISTRO_COMPONENTS##
 packages=##IMAGE_PREINSTALL##
 omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_APT##
+suite=##ISAR_DISTRO_SUITE##
+components=main
+omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index f4b0125..7798b25 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -46,6 +46,9 @@ do_rootfs() {
         -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
         -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
+        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
     # Create root filesystem
@@ -57,4 +60,5 @@ do_rootfs() {
     sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
 }
 
-addtask rootfs before do_populate after do_unpack
+addtask rootfs before do_build after do_populate
+
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
     fi
 }
 
-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 610dd74..e1a5c85 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -61,7 +61,7 @@ addtask cache_config before do_fetch
 
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 
-# Install Debian packages, that were built from sources
+# Populate Isar apt repository by newly built packages
 do_populate() {
     if [ -n "${IMAGE_INSTALL}" ]; then
         for p in ${IMAGE_INSTALL}; do
@@ -71,15 +71,6 @@ do_populate() {
                           includedeb ${DEBDISTRONAME} \
                           ${DEPLOY_DIR_DEB}/${p}_*.deb
         done
-        sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
-        for p in ${IMAGE_INSTALL}; do
-            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
-        done
-
-        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
-        sudo rm -rf ${IMAGE_ROOTFS}/deb
     fi
 }
 
-- 
2.1.4

Re: [PATCH 1/4] apt: Generate configs for apt

[Claudius Heine]

Hi,

On 10/05/2017 12:08 PM, Alexander Smirnov wrote:
> Generate reprepro configuration files for apt deployment.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>   meta-isar/conf/layer.conf                            | 10 ++++++++--
>   meta-isar/recipes-core/images/files/distributions.in |  3 +++
>   meta/classes/image.bbclass                           | 17 ++++++++++++++++-
>   3 files changed, 27 insertions(+), 3 deletions(-)
>   create mode 100644 meta-isar/recipes-core/images/files/distributions.in
> 
> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
> index 72a103c..0950a34 100644
> --- a/meta-isar/conf/layer.conf
> +++ b/meta-isar/conf/layer.conf
> @@ -1,5 +1,5 @@
>   # This software is a part of ISAR.
> -# Copyright (C) 2015-2016 ilbers GmbH
> +# Copyright (C) 2015-2017 ilbers GmbH
>   
>   # We have a conf and classes directory, add to BBPATH
>   BBPATH .= ":${LAYERDIR}"
> @@ -13,6 +13,12 @@ BBFILE_PRIORITY_isar = "5"
>   
>   # This should only be incremented on significant changes that will
>   # cause compatibility issues with other layers
> -LAYERVERSION_isar = "2"
> +LAYERVERSION_isar = "3"
>   
>   LAYERDIR_isar = "${LAYERDIR}"
> +
> +# Codename of the repository created by the caching class
> +DEBDISTRONAME = "isar"
> +
> +# Path to the Isar apt repository
> +DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"

Thanks!

I would prefer that being handled the same way the DL_DIR and SSTATE_DIR 
is handled. So the same variable naming scheme (maybe APT_CACHE_DIR) and 
similar default paths (maybe "${TOPDIR}/aptcache"). The deploy dir is 
not really the place to do work in.

If you want to have something deployed to the deploy dir then it should 
be done via a recipe. I would suggest something like a 
"do_deploy_repository_deb" task in image recipes. But thats a separate 
patch.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 1/4] apt: Generate configs for apt

[Alexander Smirnov]

On 10/05/2017 02:14 PM, Claudius Heine wrote:
> Hi,
> 
> On 10/05/2017 12:08 PM, Alexander Smirnov wrote:
>> Generate reprepro configuration files for apt deployment.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta-isar/conf/layer.conf                            | 10 ++++++++--
>>   meta-isar/recipes-core/images/files/distributions.in |  3 +++
>>   meta/classes/image.bbclass                           | 17 
>> ++++++++++++++++-
>>   3 files changed, 27 insertions(+), 3 deletions(-)
>>   create mode 100644 meta-isar/recipes-core/images/files/distributions.in
>>
>> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
>> index 72a103c..0950a34 100644
>> --- a/meta-isar/conf/layer.conf
>> +++ b/meta-isar/conf/layer.conf
>> @@ -1,5 +1,5 @@
>>   # This software is a part of ISAR.
>> -# Copyright (C) 2015-2016 ilbers GmbH
>> +# Copyright (C) 2015-2017 ilbers GmbH
>>   # We have a conf and classes directory, add to BBPATH
>>   BBPATH .= ":${LAYERDIR}"
>> @@ -13,6 +13,12 @@ BBFILE_PRIORITY_isar = "5"
>>   # This should only be incremented on significant changes that will
>>   # cause compatibility issues with other layers
>> -LAYERVERSION_isar = "2"
>> +LAYERVERSION_isar = "3"
>>   LAYERDIR_isar = "${LAYERDIR}"
>> +
>> +# Codename of the repository created by the caching class
>> +DEBDISTRONAME = "isar"
>> +
>> +# Path to the Isar apt repository
>> +DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
> 
> Thanks!
> 
> I would prefer that being handled the same way the DL_DIR and SSTATE_DIR 
> is handled. So the same variable naming scheme (maybe APT_CACHE_DIR) and 
> similar default paths (maybe "${TOPDIR}/aptcache"). The deploy dir is 
> not really the place to do work in.

This should be in deploy directory, because "${DEPLOY_DIR}/apt" folder 
contains self-contained apt repository that could be exported/published 
by Isar user's demand, like images.

> 
> If you want to have something deployed to the deploy dir then it should 
> be done via a recipe. I would suggest something like a 
> "do_deploy_repository_deb" task in image recipes. But thats a separate 
> patch.

Please see the whole series. Apt repository population is performed in 
image recipe by do_populate task.

Alex

Re: [PATCH 2/4] apt: Generate Isar reprepro database

[Claudius Heine]

Hi,

On 10/05/2017 12:08 PM, Alexander Smirnov wrote:
> Generate reprepro database for Isar apt repository.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>   meta-isar/conf/layer.conf        |  3 +++
>   meta-isar/conf/local.conf.sample |  4 ++++
>   meta/classes/image.bbclass       | 34 ++++++++++++++++++++++++++++++++++
>   3 files changed, 41 insertions(+)
> 
> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
> index 0950a34..cd42f06 100644
> --- a/meta-isar/conf/layer.conf
> +++ b/meta-isar/conf/layer.conf
> @@ -22,3 +22,6 @@ DEBDISTRONAME = "isar"
>   
>   # Path to the Isar apt repository
>   DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
> +
> +# Path to the Isar databases used by `reprepro`
> +DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"

The same comment in the previous patch applies here.

> diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
> index a456b1b..660958f 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
>   #
>   # Default parallel jobs for bitbake:
>   BB_NUMBER_THREADS = "4"
> +
> +#
> +# Number of attempts to try to get reprepro lock for access to apt cache
> +REPREPRO_LOCK_ATTEMPTS = "16"
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 57705a7..47f53a3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
>   
>   inherit ${IMAGE_TYPE}
>   
> +call_reprepro() {
> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> +        #  According to `sh` manual page, shell exit statuses fall between
> +        # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit
> +        # integer results value (239).
> +        eexist=$(python -c 'import errno; print(256-errno.EEXIST)')

You can just use the python interpreter that is already running like this:

     def _eexist():
     	import errno
     	return 265-errno.EEXIST

     ...

     eexist="${@_eexist()}"

> +        retval="0"
> +        reprepro $* || retval="$?"
> +
> +        # If reprepro has failed to get database lock, it returns EEXIST code.
> +        # In this case we continue trying to get lock until max amount of
> +        # attempts is reached.
> +        if [ $retval -eq $eexist ]; then
> +            bbwarn "Failed to get reprepro lock, trying again..."
> +            sleep 5
> +        else
> +            break
> +        fi
> +    done
> +
> +    if [ $retval -ne 0 ]; then
> +        bbfatal "reprepro failed"
> +    fi
> +}
> +
>   CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
>   do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>   do_cache_config[stamp-extra-info] = "${DISTRO}"
> @@ -21,6 +46,15 @@ do_cache_config() {
>           sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>               ${FILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions
>       fi
> +
> +    path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
> +    path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
> +
> +    if [ ! -d "${path_databases}" ]; then
> +        call_reprepro -b ${path_cache} \
> +                      --dbdir ${path_databases} \
> +                      export ${DEBDISTRONAME}
> +    fi
>   }
>   
>   addtask cache_config before do_fetch
> 

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 1/4] apt: Generate configs for apt

[Claudius Heine]

On 10/05/2017 01:43 PM, Alexander Smirnov wrote:
> On 10/05/2017 02:14 PM, Claudius Heine wrote:
>> Hi,
>>
>> On 10/05/2017 12:08 PM, Alexander Smirnov wrote:
>>> Generate reprepro configuration files for apt deployment.
>>>
>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>> ---
>>>   meta-isar/conf/layer.conf                            | 10 ++++++++--
>>>   meta-isar/recipes-core/images/files/distributions.in |  3 +++
>>>   meta/classes/image.bbclass                           | 17 
>>> ++++++++++++++++-
>>>   3 files changed, 27 insertions(+), 3 deletions(-)
>>>   create mode 100644 
>>> meta-isar/recipes-core/images/files/distributions.in
>>>
>>> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
>>> index 72a103c..0950a34 100644
>>> --- a/meta-isar/conf/layer.conf
>>> +++ b/meta-isar/conf/layer.conf
>>> @@ -1,5 +1,5 @@
>>>   # This software is a part of ISAR.
>>> -# Copyright (C) 2015-2016 ilbers GmbH
>>> +# Copyright (C) 2015-2017 ilbers GmbH
>>>   # We have a conf and classes directory, add to BBPATH
>>>   BBPATH .= ":${LAYERDIR}"
>>> @@ -13,6 +13,12 @@ BBFILE_PRIORITY_isar = "5"
>>>   # This should only be incremented on significant changes that will
>>>   # cause compatibility issues with other layers
>>> -LAYERVERSION_isar = "2"
>>> +LAYERVERSION_isar = "3"
>>>   LAYERDIR_isar = "${LAYERDIR}"
>>> +
>>> +# Codename of the repository created by the caching class
>>> +DEBDISTRONAME = "isar"
>>> +
>>> +# Path to the Isar apt repository
>>> +DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
>>
>> Thanks!
>>
>> I would prefer that being handled the same way the DL_DIR and 
>> SSTATE_DIR is handled. So the same variable naming scheme (maybe 
>> APT_CACHE_DIR) and similar default paths (maybe "${TOPDIR}/aptcache"). 
>> The deploy dir is not really the place to do work in.
> 
> This should be in deploy directory, because "${DEPLOY_DIR}/apt" folder 
> contains self-contained apt repository that could be exported/published 
> by Isar user's demand, like images.

I took a look at how OE creates repositories and its similar to the way 
its implemented here.

I am not 100% sure that it makes sense to copy it from OE in this case, 
because we are using this repo differently (as a cache) than oe (just as 
a output of the package index) but I am willing to accept this solution, 
because people coming from OE might be more familiar this way.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 0/4 v5] Isar apt deployment

[Henning Schild]

Hi,

i just tried it and the task do_cache_config was not executed before
the first do_populate.

I would suggest the following changes:
- create one task in dpkg-base.bbclass that does the following
 - config and init reprepro if no other recipe did that before
 - add package
 - use bitkages task[lockfiles] to deal with races between recipes,
   put distro into lockfile-name so we have one lock per distro 
- drop do_populate
- call the new task instead of, or in do_deploy_deb

Henning

Am Thu, 5 Oct 2017 13:08:03 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Hi all,
> 
> this series switch Isar internal binary package processing to apt
> repository. It performs this following:
> 
> 1. Create repositories 'tmp/deploy/apt' for all architectures
> requested by multiconfig.
> 
> 2. Generate reprepro database.
> 
> 3. Put all the newly built packages to apt repository.
> 
> 4. Pass this Isar repository to image multistrap.
> 
> Documentation will be updated after agreement on this implementation.
> 
> Changes since v4:
>  - Drop meta-isar-bin layer.
> 
> With best regards,
> Alex
> 
> Alexander Smirnov (4):
>   apt: Generate configs for apt
>   apt: Generate Isar reprepro database
>   apt: Populate Isar apt
>   apt: Install packages via multistrap
> 
>  meta-isar/conf/layer.conf                          | 13 ++++-
>  meta-isar/conf/local.conf.sample                   |  4 ++
>  .../recipes-core/images/files/distributions.in     |  3 +
>  .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
>  meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
>  meta/classes/ext4-img.bbclass                      |  2 +-
>  meta/classes/image.bbclass                         | 65
> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
> deletions(-) create mode 100644
> meta-isar/recipes-core/images/files/distributions.in
> 

Re: [PATCH 2/4] apt: Generate Isar reprepro database

[Henning Schild]

Am Thu, 5 Oct 2017 13:08:05 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Generate reprepro database for Isar apt repository.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta-isar/conf/layer.conf        |  3 +++
>  meta-isar/conf/local.conf.sample |  4 ++++
>  meta/classes/image.bbclass       | 34
> ++++++++++++++++++++++++++++++++++ 3 files changed, 41 insertions(+)
> 
> diff --git a/meta-isar/conf/layer.conf b/meta-isar/conf/layer.conf
> index 0950a34..cd42f06 100644
> --- a/meta-isar/conf/layer.conf
> +++ b/meta-isar/conf/layer.conf
> @@ -22,3 +22,6 @@ DEBDISTRONAME = "isar"
>  
>  # Path to the Isar apt repository
>  DEPLOY_DIR_APT ?= "${DEPLOY_DIR}/apt"
> +
> +# Path to the Isar databases used by `reprepro`
> +DEPLOY_DIR_DB ?= "${DEPLOY_DIR}/db"
> diff --git a/meta-isar/conf/local.conf.sample
> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644
> --- a/meta-isar/conf/local.conf.sample
> +++ b/meta-isar/conf/local.conf.sample
> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw"
>  #
>  # Default parallel jobs for bitbake:
>  BB_NUMBER_THREADS = "4"
> +
> +#
> +# Number of attempts to try to get reprepro lock for access to apt
> cache +REPREPRO_LOCK_ATTEMPTS = "16"
> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
> index 57705a7..47f53a3 100644
> --- a/meta/classes/image.bbclass
> +++ b/meta/classes/image.bbclass
> @@ -10,6 +10,31 @@ IMAGE_ROOTFS   = "${WORKDIR}/rootfs"
>  
>  inherit ${IMAGE_TYPE}
>  
> +call_reprepro() {
> +    for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do
> +        #  According to `sh` manual page, shell exit statuses fall
> between
> +        # 0-255. The EEXIST error code is (-17), so casting to
> usigned 8-bit
> +        # integer results value (239).
> +        eexist=$(python -c 'import errno; print(256-errno.EEXIST)')

Now that there are not magic numbers anymore we can skip the verbose
comment.

Henning

> +        retval="0"
> +        reprepro $* || retval="$?"
> +
> +        # If reprepro has failed to get database lock, it returns
> EEXIST code.
> +        # In this case we continue trying to get lock until max
> amount of
> +        # attempts is reached.
> +        if [ $retval -eq $eexist ]; then
> +            bbwarn "Failed to get reprepro lock, trying again..."
> +            sleep 5
> +        else
> +            break
> +        fi
> +    done
> +
> +    if [ $retval -ne 0 ]; then
> +        bbfatal "reprepro failed"
> +    fi
> +}
> +
>  CACHE_CONF_DIR = "${DEPLOY_DIR_APT}/${DISTRO}/conf"
>  do_cache_config[dirs] = "${CACHE_CONF_DIR}"
>  do_cache_config[stamp-extra-info] = "${DISTRO}"
> @@ -21,6 +46,15 @@ do_cache_config() {
>          sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \
>              ${FILESDIR}/distributions.in >
> ${CACHE_CONF_DIR}/distributions fi
> +
> +    path_cache="${DEPLOY_DIR_APT}/${DISTRO}"
> +    path_databases="${DEPLOY_DIR_DB}/${DISTRO}"
> +
> +    if [ ! -d "${path_databases}" ]; then
> +        call_reprepro -b ${path_cache} \
> +                      --dbdir ${path_databases} \
> +                      export ${DEBDISTRONAME}
> +    fi
>  }
>  
>  addtask cache_config before do_fetch

Re: [PATCH 0/4 v5] Isar apt deployment

[Claudius Heine]

Hi,

On 10/09/2017 02:00 PM, [ext] Henning Schild wrote:
> Hi,
> 
> i just tried it and the task do_cache_config was not executed before
> the first do_populate.

Same issue here.

Claudius

-- 
DENX Software Engineering GmbH,      Managing Director: Wolfgang Denk
HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany
Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de

Re: [PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

Hi,

> i just tried it and the task do_cache_config was not executed before
> the first do_populate.

Could you please specify more details:

  - Build command
  - Commit ID

Alex

> 
> I would suggest the following changes:
> - create one task in dpkg-base.bbclass that does the following
>   - config and init reprepro if no other recipe did that before
>   - add package
>   - use bitkages task[lockfiles] to deal with races between recipes,
>     put distro into lockfile-name so we have one lock per distro
> - drop do_populate
> - call the new task instead of, or in do_deploy_deb
> 
> Henning
> 
> Am Thu, 5 Oct 2017 13:08:03 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Hi all,
>>
>> this series switch Isar internal binary package processing to apt
>> repository. It performs this following:
>>
>> 1. Create repositories 'tmp/deploy/apt' for all architectures
>> requested by multiconfig.
>>
>> 2. Generate reprepro database.
>>
>> 3. Put all the newly built packages to apt repository.
>>
>> 4. Pass this Isar repository to image multistrap.
>>
>> Documentation will be updated after agreement on this implementation.
>>
>> Changes since v4:
>>   - Drop meta-isar-bin layer.
>>
>> With best regards,
>> Alex
>>
>> Alexander Smirnov (4):
>>    apt: Generate configs for apt
>>    apt: Generate Isar reprepro database
>>    apt: Populate Isar apt
>>    apt: Install packages via multistrap
>>
>>   meta-isar/conf/layer.conf                          | 13 ++++-
>>   meta-isar/conf/local.conf.sample                   |  4 ++
>>   .../recipes-core/images/files/distributions.in     |  3 +
>>   .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
>>   meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
>>   meta/classes/ext4-img.bbclass                      |  2 +-
>>   meta/classes/image.bbclass                         | 65
>> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
>> deletions(-) create mode 100644
>> meta-isar/recipes-core/images/files/distributions.in
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 0/4 v5] Isar apt deployment

[Henning Schild]

On Wed, 18 Oct 2017 14:06:53 +0300
Alexander Smirnov <asmirnov@ilbers.de> wrote:

> Hi,
> 
> > i just tried it and the task do_cache_config was not executed before
> > the first do_populate.  
> 
> Could you please specify more details:
> 
>   - Build command

Well i am using layering, and so is Claudius. So the command wont help
you much. It should basically boil down to

bitbake multiconfig:qemuamd64-stretch:isar-image-base

>   - Commit ID

f33e48bd07039032e3

I am guessing you can not reproduce the issue, we should find out why.
But since Claudius confirmend that he can not use these patches either,
and there are still some open points, i think you should remove those
patches from next.

Henning

> Alex
> 
> > 
> > I would suggest the following changes:
> > - create one task in dpkg-base.bbclass that does the following
> >   - config and init reprepro if no other recipe did that before
> >   - add package
> >   - use bitkages task[lockfiles] to deal with races between recipes,
> >     put distro into lockfile-name so we have one lock per distro
> > - drop do_populate
> > - call the new task instead of, or in do_deploy_deb
> > 
> > Henning
> > 
> > Am Thu, 5 Oct 2017 13:08:03 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Hi all,
> >>
> >> this series switch Isar internal binary package processing to apt
> >> repository. It performs this following:
> >>
> >> 1. Create repositories 'tmp/deploy/apt' for all architectures
> >> requested by multiconfig.
> >>
> >> 2. Generate reprepro database.
> >>
> >> 3. Put all the newly built packages to apt repository.
> >>
> >> 4. Pass this Isar repository to image multistrap.
> >>
> >> Documentation will be updated after agreement on this
> >> implementation.
> >>
> >> Changes since v4:
> >>   - Drop meta-isar-bin layer.
> >>
> >> With best regards,
> >> Alex
> >>
> >> Alexander Smirnov (4):
> >>    apt: Generate configs for apt
> >>    apt: Generate Isar reprepro database
> >>    apt: Populate Isar apt
> >>    apt: Install packages via multistrap
> >>
> >>   meta-isar/conf/layer.conf                          | 13 ++++-
> >>   meta-isar/conf/local.conf.sample                   |  4 ++
> >>   .../recipes-core/images/files/distributions.in     |  3 +
> >>   .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
> >>   meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
> >>   meta/classes/ext4-img.bbclass                      |  2 +-
> >>   meta/classes/image.bbclass                         | 65
> >> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
> >> deletions(-) create mode 100644
> >> meta-isar/recipes-core/images/files/distributions.in
> >>  
> >   
> 

Re: [PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

On 10/18/2017 04:44 PM, Henning Schild wrote:
> On Wed, 18 Oct 2017 14:06:53 +0300
> Alexander Smirnov <asmirnov@ilbers.de> wrote:
> 
>> Hi,
>>
>>> i just tried it and the task do_cache_config was not executed before
>>> the first do_populate.
>>
>> Could you please specify more details:
>>
>>    - Build command
> 
> Well i am using layering, and so is Claudius. So the command wont help
> you much. It should basically boil down to
> 
> bitbake multiconfig:qemuamd64-stretch:isar-image-base
> 
>>    - Commit ID
> 
> f33e48bd07039032e3
> 
> I am guessing you can not reproduce the issue, we should find out why.
> But since Claudius confirmend that he can not use these patches either,
> and there are still some open points, i think you should remove those
> patches from next.
> 

At first, as I understand you use private layering. So if something 
doesn't work in your custom environment, without details and analysis 
it's not the case to reset 'next' branch.

Before perform any merge, apart from my local builds, the reference 
build on CI machine is also passed. So the upstream Isar does support 
the claimed features.

If I'm not able to reproduce your issue and there is no information 
about problem, then I'm not able to fix it. So if you need assistance in 
resolving the issue, please provide more details. If the problem will be 
found, the fix should be sent.

Regarding new open points appeared after v5, they contain several 
proposals what to try, but they doesn't influence on apt feature.

I'll try the things you proposed with multiconfig and if they work, I'll 
send updates for current apt implementation. But working feature in 
upstream should not be blocked.

Alex

> Henning
> 
>> Alex
>>
>>>
>>> I would suggest the following changes:
>>> - create one task in dpkg-base.bbclass that does the following
>>>    - config and init reprepro if no other recipe did that before
>>>    - add package
>>>    - use bitkages task[lockfiles] to deal with races between recipes,
>>>      put distro into lockfile-name so we have one lock per distro
>>> - drop do_populate
>>> - call the new task instead of, or in do_deploy_deb
>>>
>>> Henning
>>>
>>> Am Thu, 5 Oct 2017 13:08:03 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Hi all,
>>>>
>>>> this series switch Isar internal binary package processing to apt
>>>> repository. It performs this following:
>>>>
>>>> 1. Create repositories 'tmp/deploy/apt' for all architectures
>>>> requested by multiconfig.
>>>>
>>>> 2. Generate reprepro database.
>>>>
>>>> 3. Put all the newly built packages to apt repository.
>>>>
>>>> 4. Pass this Isar repository to image multistrap.
>>>>
>>>> Documentation will be updated after agreement on this
>>>> implementation.
>>>>
>>>> Changes since v4:
>>>>    - Drop meta-isar-bin layer.
>>>>
>>>> With best regards,
>>>> Alex
>>>>
>>>> Alexander Smirnov (4):
>>>>     apt: Generate configs for apt
>>>>     apt: Generate Isar reprepro database
>>>>     apt: Populate Isar apt
>>>>     apt: Install packages via multistrap
>>>>
>>>>    meta-isar/conf/layer.conf                          | 13 ++++-
>>>>    meta-isar/conf/local.conf.sample                   |  4 ++
>>>>    .../recipes-core/images/files/distributions.in     |  3 +
>>>>    .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
>>>>    meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
>>>>    meta/classes/ext4-img.bbclass                      |  2 +-
>>>>    meta/classes/image.bbclass                         | 65
>>>> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
>>>> deletions(-) create mode 100644
>>>> meta-isar/recipes-core/images/files/distributions.in
>>>>   
>>>    
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

[-- Attachment #1: Type: text/plain, Size: 4102 bytes --]



On 10/18/2017 08:10 PM, Alexander Smirnov wrote:
> On 10/18/2017 04:44 PM, Henning Schild wrote:
>> On Wed, 18 Oct 2017 14:06:53 +0300
>> Alexander Smirnov <asmirnov@ilbers.de> wrote:
>>
>>> Hi,
>>>
>>>> i just tried it and the task do_cache_config was not executed before
>>>> the first do_populate.
>>>
>>> Could you please specify more details:
>>>
>>>    - Build command
>>
>> Well i am using layering, and so is Claudius. So the command wont help
>> you much. It should basically boil down to
>>
>> bitbake multiconfig:qemuamd64-stretch:isar-image-base
>>
>>>    - Commit ID
>>
>> f33e48bd07039032e3
>>
>> I am guessing you can not reproduce the issue, we should find out why.
>> But since Claudius confirmend that he can not use these patches either,
>> and there are still some open points, i think you should remove those
>> patches from next.
>>
> 
> At first, as I understand you use private layering. So if something 
> doesn't work in your custom environment, without details and analysis 
> it's not the case to reset 'next' branch.
> 
> Before perform any merge, apart from my local builds, the reference 
> build on CI machine is also passed. So the upstream Isar does support 
> the claimed features.
> 
> If I'm not able to reproduce your issue and there is no information 
> about problem, then I'm not able to fix it. So if you need assistance in 
> resolving the issue, please provide more details. If the problem will be 
> found, the fix should be sent.

Please try the patch attached.

> 
> Regarding new open points appeared after v5, they contain several 
> proposals what to try, but they doesn't influence on apt feature.
> 
> I'll try the things you proposed with multiconfig and if they work, I'll 
> send updates for current apt implementation. But working feature in 
> upstream should not be blocked.
> > Alex
> 
>> Henning
>>
>>> Alex
>>>
>>>>
>>>> I would suggest the following changes:
>>>> - create one task in dpkg-base.bbclass that does the following
>>>>    - config and init reprepro if no other recipe did that before
>>>>    - add package
>>>>    - use bitkages task[lockfiles] to deal with races between recipes,
>>>>      put distro into lockfile-name so we have one lock per distro
>>>> - drop do_populate
>>>> - call the new task instead of, or in do_deploy_deb
>>>>
>>>> Henning
>>>>
>>>> Am Thu, 5 Oct 2017 13:08:03 +0300
>>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>>> Hi all,
>>>>>
>>>>> this series switch Isar internal binary package processing to apt
>>>>> repository. It performs this following:
>>>>>
>>>>> 1. Create repositories 'tmp/deploy/apt' for all architectures
>>>>> requested by multiconfig.
>>>>>
>>>>> 2. Generate reprepro database.
>>>>>
>>>>> 3. Put all the newly built packages to apt repository.
>>>>>
>>>>> 4. Pass this Isar repository to image multistrap.
>>>>>
>>>>> Documentation will be updated after agreement on this
>>>>> implementation.
>>>>>
>>>>> Changes since v4:
>>>>>    - Drop meta-isar-bin layer.
>>>>>
>>>>> With best regards,
>>>>> Alex
>>>>>
>>>>> Alexander Smirnov (4):
>>>>>     apt: Generate configs for apt
>>>>>     apt: Generate Isar reprepro database
>>>>>     apt: Populate Isar apt
>>>>>     apt: Install packages via multistrap
>>>>>
>>>>>    meta-isar/conf/layer.conf                          | 13 ++++-
>>>>>    meta-isar/conf/local.conf.sample                   |  4 ++
>>>>>    .../recipes-core/images/files/distributions.in     |  3 +
>>>>>    .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
>>>>>    meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
>>>>>    meta/classes/ext4-img.bbclass                      |  2 +-
>>>>>    meta/classes/image.bbclass                         | 65
>>>>> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
>>>>> deletions(-) create mode 100644
>>>>> meta-isar/recipes-core/images/files/distributions.in
>>>
>>
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

[-- Attachment #2: apt_fix.patch --]
[-- Type: text/x-patch, Size: 355 bytes --]

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2284655..42386a6 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -54,7 +54,7 @@ do_cache_config() {
     fi
 }
 
-addtask cache_config before do_fetch
+addtask cache_config before do_populate
 
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 

Re: [PATCH 0/4 v5] Isar apt deployment

[Henning Schild]

On Wed, 18 Oct 2017 20:10:49 +0300
Alexander Smirnov <asmirnov@ilbers.de> wrote:

> On 10/18/2017 04:44 PM, Henning Schild wrote:
> > On Wed, 18 Oct 2017 14:06:53 +0300
> > Alexander Smirnov <asmirnov@ilbers.de> wrote:
> >   
> >> Hi,
> >>  
> >>> i just tried it and the task do_cache_config was not executed
> >>> before the first do_populate.  
> >>
> >> Could you please specify more details:
> >>
> >>    - Build command  
> > 
> > Well i am using layering, and so is Claudius. So the command wont
> > help you much. It should basically boil down to
> > 
> > bitbake multiconfig:qemuamd64-stretch:isar-image-base
> >   
> >>    - Commit ID  
> > 
> > f33e48bd07039032e3
> > 
> > I am guessing you can not reproduce the issue, we should find out
> > why. But since Claudius confirmend that he can not use these
> > patches either, and there are still some open points, i think you
> > should remove those patches from next.
> >   
> 
> At first, as I understand you use private layering. So if something 
> doesn't work in your custom environment, without details and analysis 
> it's not the case to reset 'next' branch.
> 
> Before perform any merge, apart from my local builds, the reference 
> build on CI machine is also passed. So the upstream Isar does support 
> the claimed features.
> 
> If I'm not able to reproduce your issue and there is no information 
> about problem, then I'm not able to fix it. So if you need assistance
> in resolving the issue, please provide more details. If the problem
> will be found, the fix should be sent.
> 
> Regarding new open points appeared after v5, they contain several 
> proposals what to try, but they doesn't influence on apt feature.
> 
> I'll try the things you proposed with multiconfig and if they work,
> I'll send updates for current apt implementation. But working feature
> in upstream should not be blocked.

I will try to provide you with a way to reproduce the issue and will
look at the patch you attached. Still the whole series went into next
while it was still under review and still had open points. As far as i
can tell the only difference to my build is that BB_NUMBER_THREADS was
1, but i might be missing something here.
That said i do not understand why you refuse to take it back out, i can
not work on next since these patches are in and i can not even test the
new stuff that was merged on top, same might be true for other
developers.
Also the next q is getting pretty long, as far as i am concerned
everything up to 9c6e8a1e2fb6b65 can be rebased on master and master
can be advanced to that point.

Henning

> Alex
> 
> > Henning
> >   
> >> Alex
> >>  
> >>>
> >>> I would suggest the following changes:
> >>> - create one task in dpkg-base.bbclass that does the following
> >>>    - config and init reprepro if no other recipe did that before
> >>>    - add package
> >>>    - use bitkages task[lockfiles] to deal with races between
> >>> recipes, put distro into lockfile-name so we have one lock per
> >>> distro
> >>> - drop do_populate
> >>> - call the new task instead of, or in do_deploy_deb
> >>>
> >>> Henning
> >>>
> >>> Am Thu, 5 Oct 2017 13:08:03 +0300
> >>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >>>      
> >>>> Hi all,
> >>>>
> >>>> this series switch Isar internal binary package processing to apt
> >>>> repository. It performs this following:
> >>>>
> >>>> 1. Create repositories 'tmp/deploy/apt' for all architectures
> >>>> requested by multiconfig.
> >>>>
> >>>> 2. Generate reprepro database.
> >>>>
> >>>> 3. Put all the newly built packages to apt repository.
> >>>>
> >>>> 4. Pass this Isar repository to image multistrap.
> >>>>
> >>>> Documentation will be updated after agreement on this
> >>>> implementation.
> >>>>
> >>>> Changes since v4:
> >>>>    - Drop meta-isar-bin layer.
> >>>>
> >>>> With best regards,
> >>>> Alex
> >>>>
> >>>> Alexander Smirnov (4):
> >>>>     apt: Generate configs for apt
> >>>>     apt: Generate Isar reprepro database
> >>>>     apt: Populate Isar apt
> >>>>     apt: Install packages via multistrap
> >>>>
> >>>>    meta-isar/conf/layer.conf                          | 13 ++++-
> >>>>    meta-isar/conf/local.conf.sample                   |  4 ++
> >>>>    .../recipes-core/images/files/distributions.in     |  3 +
> >>>>    .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
> >>>>    meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
> >>>>    meta/classes/ext4-img.bbclass                      |  2 +-
> >>>>    meta/classes/image.bbclass                         | 65
> >>>> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
> >>>> deletions(-) create mode 100644
> >>>> meta-isar/recipes-core/images/files/distributions.in
> >>>>     
> >>>      
> >>  
> >   
> 

Re: [PATCH 0/4 v5] Isar apt deployment

[Baurzhan Ismagulov]

On Thu, Oct 19, 2017 at 10:41:45AM +0200, Henning Schild wrote:
> I will try to provide you with a way to reproduce the issue and will
> look at the patch you attached.

That would be great. You don't even need to spend time providing a minimal use
case; I suggest that you provide your tree(s) as is off-list and we look at
that with high priority.


> Still the whole series went into next while it was still under review and
> still had open points.

The feature was in discussion since 2017-08-27. The last open points -- EEXIST
and meta-isar-bin -- were addressed in v5. After no comments followed, the
series was merged. It's after that that you report a build problem with an
unpublished layer and suggest a completely new architecture.

Under these circumstances, my suggestion is to make use of the existing feature
v1 -- CIP and other people need that -- and implement the new architecture in
v2.


> That said i do not understand why you refuse to take it back out, i can
> not work on next since these patches are in and i can not even test the
> new stuff that was merged on top, same might be true for other
> developers.

Because the series works for all use cases known to us.

You are demanding to block the feature till you resolve the problem. But the
win-win solution is to solve your problem. This is what we should aim at. We
get mails from you since ten days, but what we would like to have is 1. your
tree, as is, and 2. feedback to Alex's patch -- I see that as a homework before
demanding the same for the third time.


> Also the next q is getting pretty long

We had intended to do a merge to master in Sep, but failed to do so. Merging
the queues while short is exactly what we are trying to achieve right now.


With kind regards,
Baurzhan.

Re: [PATCH 0/4 v5] Isar apt deployment

[Henning Schild]

On Thu, 19 Oct 2017 11:54:25 +0200
Baurzhan Ismagulov <ibr@radix50.net> wrote:

> On Thu, Oct 19, 2017 at 10:41:45AM +0200, Henning Schild wrote:
> > I will try to provide you with a way to reproduce the issue and will
> > look at the patch you attached.  
> 
> That would be great. You don't even need to spend time providing a
> minimal use case; I suggest that you provide your tree(s) as is
> off-list and we look at that with high priority.

I will have to look into that, one way or another i can not just give
you what i have.

> > Still the whole series went into next while it was still under
> > review and still had open points.  
> 
> The feature was in discussion since 2017-08-27. The last open points
> -- EEXIST and meta-isar-bin -- were addressed in v5. After no
> comments followed, the series was merged. It's after that that you
> report a build problem with an unpublished layer and suggest a
> completely new architecture.

The series was merged 3h after it was presented for review, my bad for
being so slow.
Issues not found in earlier rounds of reviews are still issues. And i
have heard many comments about that weird retry-loop, unfortunately not
everyone is active on the list.

> Under these circumstances, my suggestion is to make use of the
> existing feature v1 -- CIP and other people need that -- and
> implement the new architecture in v2.

It is not merged yet, it is just in a rebasing branch where you can
just drop it! But if you want to keep a known to be broken v1 go ahead.

> > That said i do not understand why you refuse to take it back out, i
> > can not work on next since these patches are in and i can not even
> > test the new stuff that was merged on top, same might be true for
> > other developers.  
> 
> Because the series works for all use cases known to us.

I would like to understand how the code expresses that do_cache_config
has to happen before the first do_populate.
As far as i can see it does not do that, so the race is obvious, but
might be hard to reproduce. My first guess is that BB_NUMBER_THREADS=1
might help get you there as well.
However if it is expressed, i will do my homework and find why that
expression does not work in my case.
 
> You are demanding to block the feature till you resolve the problem.
> But the win-win solution is to solve your problem. This is what we
> should aim at. We get mails from you since ten days, but what we
> would like to have is 1. your tree, as is, and 2. feedback to Alex's
> patch -- I see that as a homework before demanding the same for the
> third time.

1. only works in our intranet, sorry
2. i have provided that feedback several times and suggested
[lockfiles] and [depends] as means to deal with races and express deps

Henning

> 
> > Also the next q is getting pretty long  
> 
> We had intended to do a merge to master in Sep, but failed to do so.
> Merging the queues while short is exactly what we are trying to
> achieve right now.
> 
> 
> With kind regards,
> Baurzhan.
> 

Re: [PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

[-- Attachment #1: Type: text/plain, Size: 4032 bytes --]

On 10/19/2017 04:30 PM, Henning Schild wrote:
> On Thu, 19 Oct 2017 11:54:25 +0200
> Baurzhan Ismagulov <ibr@radix50.net> wrote:
> 
>> On Thu, Oct 19, 2017 at 10:41:45AM +0200, Henning Schild wrote:
>>> I will try to provide you with a way to reproduce the issue and will
>>> look at the patch you attached.
>>
>> That would be great. You don't even need to spend time providing a
>> minimal use case; I suggest that you provide your tree(s) as is
>> off-list and we look at that with high priority.
> 
> I will have to look into that, one way or another i can not just give
> you what i have.
> 
>>> Still the whole series went into next while it was still under
>>> review and still had open points.
>>
>> The feature was in discussion since 2017-08-27. The last open points
>> -- EEXIST and meta-isar-bin -- were addressed in v5. After no
>> comments followed, the series was merged. It's after that that you
>> report a build problem with an unpublished layer and suggest a
>> completely new architecture.
> 
> The series was merged 3h after it was presented for review, my bad for
> being so slow.

The series was sent to the list 05.10.2017 at 12:00 CES. Series was 
pushed to upstream 06.10.2017 after 17:00 CES (don't remember exact 
local time, sorry). I suppose you look at commit dates, which reflect 
the time I call 'git commit' on my machine and have nothing common with 
pushing.

So please be more correct next time.

> Issues not found in earlier rounds of reviews are still issues. And i
> have heard many comments about that weird retry-loop, unfortunately not
> everyone is active on the list.

All the issues found in v4 were fixed, features supported by clean Isar 
were tested and eventually series was published. If later you propose to 
try another way to do the same things, I'll try it and come back with 
the results. But upstream tree should not be frozen until all your ideas 
and proposals will be evaluated.

> 
>> Under these circumstances, my suggestion is to make use of the
>> existing feature v1 -- CIP and other people need that -- and
>> implement the new architecture in v2.
> 
> It is not merged yet, it is just in a rebasing branch where you can
> just drop it! But if you want to keep a known to be broken v1 go ahead.
> 
>>> That said i do not understand why you refuse to take it back out, i
>>> can not work on next since these patches are in and i can not even
>>> test the new stuff that was merged on top, same might be true for
>>> other developers.
>>
>> Because the series works for all use cases known to us.
> 
> I would like to understand how the code expresses that do_cache_config
> has to happen before the first do_populate.
> As far as i can see it does not do that, so the race is obvious, but
> might be hard to reproduce. My first guess is that BB_NUMBER_THREADS=1
> might help get you there as well.
> However if it is expressed, i will do my homework and find why that
> expression does not work in my case.
>   

I've listed the tasks chain and noticed that do_populate has no 
predecessor what could be the source of your issue. Please confirm if 
the change attached fixes your issue while I'll update the respective 
patch in v5 series.

Alex

>> You are demanding to block the feature till you resolve the problem.
>> But the win-win solution is to solve your problem. This is what we
>> should aim at. We get mails from you since ten days, but what we
>> would like to have is 1. your tree, as is, and 2. feedback to Alex's
>> patch -- I see that as a homework before demanding the same for the
>> third time.
> 
> 1. only works in our intranet, sorry
> 2. i have provided that feedback several times and suggested
> [lockfiles] and [depends] as means to deal with races and express deps
> 
> Henning
> 
>>
>>> Also the next q is getting pretty long
>>
>> We had intended to do a merge to master in Sep, but failed to do so.
>> Merging the queues while short is exactly what we are trying to
>> achieve right now.
>>
>>
>> With kind regards,
>> Baurzhan.
>>
> 

[-- Attachment #2: populate.patch --]
[-- Type: text/x-patch, Size: 362 bytes --]

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2284655..3a22b2a 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -71,7 +71,7 @@ do_populate() {
     fi
 }
 
-addtask populate before do_build
+addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
 
 do_copy_boot_files() {

[PATCH 4/4 v6] apt: Install packages via multistrap

[Alexander Smirnov]

Install custom packages to target image filesystem using multistrap.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/files/multistrap.conf.in |  9 ++++++++-
 meta-isar/recipes-core/images/isar-image-base.bb       |  5 ++++-
 meta/classes/ext4-img.bbclass                          |  2 +-
 meta/classes/image.bbclass                             | 13 ++-----------
 4 files changed, 15 insertions(+), 14 deletions(-)

diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in
index 2d08c36..491e5a2 100644
--- a/meta-isar/recipes-core/images/files/multistrap.conf.in
+++ b/meta-isar/recipes-core/images/files/multistrap.conf.in
@@ -4,7 +4,7 @@
 [General]
 noauth=true
 unpack=true
-bootstrap=##DISTRO##
+bootstrap=##DISTRO## Isar
 aptsources=##DISTRO##
 configscript=##CONFIG_SCRIPT##
 setupscript=##SETUP_SCRIPT##
@@ -16,3 +16,10 @@ suite=##DISTRO_SUITE##
 components=##DISTRO_COMPONENTS##
 packages=##IMAGE_PREINSTALL##
 omitdebsrc=true
+
+[Isar]
+packages=##IMAGE_INSTALL##
+source=##DEPLOY_DIR_APT##
+suite=##ISAR_DISTRO_SUITE##
+components=main
+omitdebsrc=true
diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index f4b0125..c9e504e 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -46,6 +46,9 @@ do_rootfs() {
         -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \
         -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \
         -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \
+        -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \
+        -e 's|##DEPLOY_DIR_APT##|copy:///${DEPLOY_DIR_APT}/${DISTRO}|g' \
+        -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \
            "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf"
 
     # Create root filesystem
@@ -57,4 +60,4 @@ do_rootfs() {
     sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
 }
 
-addtask rootfs before do_populate after do_unpack
+addtask rootfs before do_build after do_populate
diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass
index 3e3768b..9fa9aa5 100644
--- a/meta/classes/ext4-img.bbclass
+++ b/meta/classes/ext4-img.bbclass
@@ -34,4 +34,4 @@ do_ext4_image() {
     fi
 }
 
-addtask ext4_image before do_build after do_populate
+addtask ext4_image before do_build after do_rootfs
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 4d3e0a6..82877f4 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -61,7 +61,7 @@ addtask cache_config before do_fetch
 
 do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}"
 
-# Install Debian packages, that were built from sources
+# Populate Isar apt repository by newly built packages
 do_populate() {
     if [ -n "${IMAGE_INSTALL}" ]; then
         for p in ${IMAGE_INSTALL}; do
@@ -71,17 +71,8 @@ do_populate() {
                           includedeb ${DEBDISTRONAME} \
                           ${DEPLOY_DIR_DEB}/${p}_*.deb
         done
-        sudo mkdir -p ${IMAGE_ROOTFS}/deb
-
-        for p in ${IMAGE_INSTALL}; do
-            sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb
-        done
-
-        sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb
-
-        sudo rm -rf ${IMAGE_ROOTFS}/deb
     fi
 }
 
-addtask populate before do_build
+addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
-- 
2.1.4

Re: [PATCH 0/4 v5] Isar apt deployment

[Henning Schild]

On Wed, 18 Oct 2017 20:14:18 +0300
Alexander Smirnov <asmirnov@ilbers.de> wrote:

> On 10/18/2017 08:10 PM, Alexander Smirnov wrote:
> > On 10/18/2017 04:44 PM, Henning Schild wrote:  
> >> On Wed, 18 Oct 2017 14:06:53 +0300
> >> Alexander Smirnov <asmirnov@ilbers.de> wrote:
> >>  
> >>> Hi,
> >>>  
> >>>> i just tried it and the task do_cache_config was not executed
> >>>> before the first do_populate.  
> >>>
> >>> Could you please specify more details:
> >>>
> >>>    - Build command  
> >>
> >> Well i am using layering, and so is Claudius. So the command wont
> >> help you much. It should basically boil down to
> >>
> >> bitbake multiconfig:qemuamd64-stretch:isar-image-base
> >>  
> >>>    - Commit ID  
> >>
> >> f33e48bd07039032e3
> >>
> >> I am guessing you can not reproduce the issue, we should find out
> >> why. But since Claudius confirmend that he can not use these
> >> patches either, and there are still some open points, i think you
> >> should remove those patches from next.
> >>  
> > 
> > At first, as I understand you use private layering. So if something 
> > doesn't work in your custom environment, without details and
> > analysis it's not the case to reset 'next' branch.
> > 
> > Before perform any merge, apart from my local builds, the reference 
> > build on CI machine is also passed. So the upstream Isar does
> > support the claimed features.
> > 
> > If I'm not able to reproduce your issue and there is no information 
> > about problem, then I'm not able to fix it. So if you need
> > assistance in resolving the issue, please provide more details. If
> > the problem will be found, the fix should be sent.  
> 
> Please try the patch attached.

That solved the problem but introduced the new problem that do_rootfs
could start before do_fetch/unpack.

See my fix here:
https://github.com/henning-schild-work/isar/commit/1e1092be974dbffbf87ab943031c44481ac0908c

btw, that branch is all of current next with only that one fix and the
two kernel version detection patches reverted. So you can consider
current next tested by me to that degree.

Now that it is not broken anymore we can delay the additional changes
for the apt stuff to a v2, IMHO..

Henning

> > 
> > Regarding new open points appeared after v5, they contain several 
> > proposals what to try, but they doesn't influence on apt feature.
> > 
> > I'll try the things you proposed with multiconfig and if they work,
> > I'll send updates for current apt implementation. But working
> > feature in upstream should not be blocked.  
> > > Alex  
> >   
> >> Henning
> >>  
> >>> Alex
> >>>  
> >>>>
> >>>> I would suggest the following changes:
> >>>> - create one task in dpkg-base.bbclass that does the following
> >>>>    - config and init reprepro if no other recipe did that before
> >>>>    - add package
> >>>>    - use bitkages task[lockfiles] to deal with races between
> >>>> recipes, put distro into lockfile-name so we have one lock per
> >>>> distro
> >>>> - drop do_populate
> >>>> - call the new task instead of, or in do_deploy_deb
> >>>>
> >>>> Henning
> >>>>
> >>>> Am Thu, 5 Oct 2017 13:08:03 +0300
> >>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:  
> >>>>> Hi all,
> >>>>>
> >>>>> this series switch Isar internal binary package processing to
> >>>>> apt repository. It performs this following:
> >>>>>
> >>>>> 1. Create repositories 'tmp/deploy/apt' for all architectures
> >>>>> requested by multiconfig.
> >>>>>
> >>>>> 2. Generate reprepro database.
> >>>>>
> >>>>> 3. Put all the newly built packages to apt repository.
> >>>>>
> >>>>> 4. Pass this Isar repository to image multistrap.
> >>>>>
> >>>>> Documentation will be updated after agreement on this
> >>>>> implementation.
> >>>>>
> >>>>> Changes since v4:
> >>>>>    - Drop meta-isar-bin layer.
> >>>>>
> >>>>> With best regards,
> >>>>> Alex
> >>>>>
> >>>>> Alexander Smirnov (4):
> >>>>>     apt: Generate configs for apt
> >>>>>     apt: Generate Isar reprepro database
> >>>>>     apt: Populate Isar apt
> >>>>>     apt: Install packages via multistrap
> >>>>>
> >>>>>    meta-isar/conf/layer.conf                          | 13 ++++-
> >>>>>    meta-isar/conf/local.conf.sample                   |  4 ++
> >>>>>    .../recipes-core/images/files/distributions.in     |  3 +
> >>>>>    .../recipes-core/images/files/multistrap.conf.in   |  9 ++-
> >>>>>    meta-isar/recipes-core/images/isar-image-base.bb   |  6 +-
> >>>>>    meta/classes/ext4-img.bbclass                      |  2 +-
> >>>>>    meta/classes/image.bbclass                         | 65
> >>>>> +++++++++++++++++++--- 7 files changed, 88 insertions(+), 14
> >>>>> deletions(-) create mode 100644
> >>>>> meta-isar/recipes-core/images/files/distributions.in  
> >>>  
> >>  
> >   
> 

Re: [PATCH 0/4 v5] Isar apt deployment

[Alexander Smirnov]

On 10/19/2017 07:16 PM, Henning Schild wrote:
> On Wed, 18 Oct 2017 20:14:18 +0300
> Alexander Smirnov <asmirnov@ilbers.de> wrote:
> 
>> On 10/18/2017 08:10 PM, Alexander Smirnov wrote:
>>> On 10/18/2017 04:44 PM, Henning Schild wrote:
>>>> On Wed, 18 Oct 2017 14:06:53 +0300
>>>> Alexander Smirnov <asmirnov@ilbers.de> wrote:
>>>>   
>>>>> Hi,
>>>>>   
>>>>>> i just tried it and the task do_cache_config was not executed
>>>>>> before the first do_populate.
>>>>>
>>>>> Could you please specify more details:
>>>>>
>>>>>     - Build command
>>>>
>>>> Well i am using layering, and so is Claudius. So the command wont
>>>> help you much. It should basically boil down to
>>>>
>>>> bitbake multiconfig:qemuamd64-stretch:isar-image-base
>>>>   
>>>>>     - Commit ID
>>>>
>>>> f33e48bd07039032e3
>>>>
>>>> I am guessing you can not reproduce the issue, we should find out
>>>> why. But since Claudius confirmend that he can not use these
>>>> patches either, and there are still some open points, i think you
>>>> should remove those patches from next.
>>>>   
>>>
>>> At first, as I understand you use private layering. So if something
>>> doesn't work in your custom environment, without details and
>>> analysis it's not the case to reset 'next' branch.
>>>
>>> Before perform any merge, apart from my local builds, the reference
>>> build on CI machine is also passed. So the upstream Isar does
>>> support the claimed features.
>>>
>>> If I'm not able to reproduce your issue and there is no information
>>> about problem, then I'm not able to fix it. So if you need
>>> assistance in resolving the issue, please provide more details. If
>>> the problem will be found, the fix should be sent.
>>
>> Please try the patch attached.
> 
> That solved the problem but introduced the new problem that do_rootfs
> could start before do_fetch/unpack.
> 
> See my fix here:
> https://github.com/henning-schild-work/isar/commit/1e1092be974dbffbf87ab943031c44481ac0908c
> 

Merged to v5, thanks.

Alex

Re: [PATCH 0/4 v5] Isar apt deployment

[Christian Storm]

> [...]
> I would suggest the following changes:
> - create one task in dpkg-base.bbclass that does the following
>  - config and init reprepro if no other recipe did that before
>  - add package
>  - use bitkages task[lockfiles] to deal with races between recipes,
>    put distro into lockfile-name so we have one lock per distro 
> - drop do_populate
> - call the new task instead of, or in do_deploy_deb

This would be quite useful to source builds with dependencies: Consider
three source built packages A, B, and C. A and B are -dev packages to be
built prior to C. Both may have dependencies satisfiable from the
Debian mirrors. Then, when building C, A and B have to be installed as
build-time dependencies into the buildchroot prior to building C. For
this, the DEPENDS= may be used in the recipe of C to install A and B
just like with any other dependencies coming from the Debian mirrors
as they're in the reprepro repository that must be made accessible in the
buildchroot. For this to work, there's an apt get update needed to
refresh the "view" of the reprepro in the buildchroot after package
addition. This update should be restricted to the reprepro and not
update from the Debian mirrors.

Currently, installing A and B into the buildchroot may be done via
via dpkg -i from DEPLOY_DEB_DIR/{A,B} but this doesn't handle
dependencies. Having A and B in the reprepro repository, and this
reprepro repository being available to the buildchroot, an apt-get
install {A,B} derived from DEPENDS= would work.



Kind regards,
   Christian

-- 
Dr. Christian Storm
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Otto-Hahn-Ring 6, 81739 M�nchen, Germany