* [PATCH v4 0/4] Basic binary cache implementation @ 2017-10-02 15:45 Alexander Smirnov 2017-10-02 15:45 ` [PATCH 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov ` (4 more replies) 0 siblings, 5 replies; 19+ messages in thread From: Alexander Smirnov @ 2017-10-02 15:45 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Hello everybody, this series introduces basic binary caching for Isar. There is a new layer: meta-isar-bin which is intended to be the binary cache. All the packages that are built in Isar are stored in this cache using reprepro. Having the separate layer makes possible to manage this cache separately from the Isar build tree. So once you have built your packages, you could re-use this cache for further builds. Also this cache repo is now used to generate image filesystem via multistrap. Changes since v3: - Check return code from reprepro, continue loop only if EEXIST code is returned. Othrwise build is aborted. Changes since v2: - Droped anonymous functions - Added number of attempts to access apt cache before fail Changes since v1: - Total refactoring - Split huge patches into small pieces - Avoid races With best regards, Alex Alexander Smirnov (4): meta-isar-bin: Add reprepro configs meta-isar-bin: Generate cache repos meta-isar-bin: Populate cache meta-isar-bin: Install packages via multistrap meta-isar-bin/conf/layer.conf | 14 +++++ meta-isar-bin/files/distributions.in | 3 ++ meta-isar/conf/bblayers.conf.sample | 1 + meta-isar/conf/local.conf.sample | 4 ++ .../recipes-core/images/files/multistrap.conf.in | 9 +++- meta-isar/recipes-core/images/isar-image-base.bb | 5 +- meta/classes/ext4-img.bbclass | 2 +- meta/classes/image.bbclass | 62 +++++++++++++++++++--- 8 files changed, 89 insertions(+), 11 deletions(-) create mode 100644 meta-isar-bin/conf/layer.conf create mode 100644 meta-isar-bin/files/distributions.in -- 2.1.4 ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 1/4] meta-isar-bin: Add reprepro configs 2017-10-02 15:45 [PATCH v4 0/4] Basic binary cache implementation Alexander Smirnov @ 2017-10-02 15:45 ` Alexander Smirnov 2017-10-02 15:45 ` [PATCH 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov ` (3 subsequent siblings) 4 siblings, 0 replies; 19+ messages in thread From: Alexander Smirnov @ 2017-10-02 15:45 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Add template for reprepro configuration file and function that generates final config for specific deistro. Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta-isar-bin/conf/layer.conf | 11 +++++++++++ meta-isar-bin/files/distributions.in | 3 +++ meta-isar/conf/bblayers.conf.sample | 1 + meta/classes/image.bbclass | 15 +++++++++++++++ 4 files changed, 30 insertions(+) create mode 100644 meta-isar-bin/conf/layer.conf create mode 100644 meta-isar-bin/files/distributions.in diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf new file mode 100644 index 0000000..3518184 --- /dev/null +++ b/meta-isar-bin/conf/layer.conf @@ -0,0 +1,11 @@ +# This software is a part of ISAR. +# Copyright (C) 2017 ilbers GmbH + +# Codename of the repository created by the caching class +DEBDISTRONAME = "isar" + +# Path to the caching repository +DEBCACHEDIR ?= "${LAYERDIR}/apt" + +# Path to the configuration files templates used by `reprepro` +DEBFILESDIR ?= "${LAYERDIR}/files" diff --git a/meta-isar-bin/files/distributions.in b/meta-isar-bin/files/distributions.in new file mode 100644 index 0000000..cd214c6 --- /dev/null +++ b/meta-isar-bin/files/distributions.in @@ -0,0 +1,3 @@ +Codename: {DISTRO_NAME} +Architectures: i386 armhf amd64 source +Components: main diff --git a/meta-isar/conf/bblayers.conf.sample b/meta-isar/conf/bblayers.conf.sample index 80867e7..53a362b 100644 --- a/meta-isar/conf/bblayers.conf.sample +++ b/meta-isar/conf/bblayers.conf.sample @@ -8,6 +8,7 @@ BBFILES ?= "" BBLAYERS ?= " \ ##ISARROOT##/meta \ ##ISARROOT##/meta-isar \ + ##ISARROOT##/meta-isar-bin \ " BBLAYERS_NON_REMOVABLE ?= " \ ##ISARROOT##/meta \ diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index 5bf9524..d30b139 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -10,6 +10,21 @@ IMAGE_ROOTFS = "${WORKDIR}/rootfs" inherit ${IMAGE_TYPE} +CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf" +do_cache_config[dirs] = "${CACHE_CONF_DIR}" +do_cache_config[stamp-extra-info] = "${DISTRO}" + +# Generate reprepro config for current distro if it doesn't exist. Once it's +# generated, this task should do nothing. +do_cache_config() { + if [ ! -e "${CACHE_CONF_DIR}/distributions" ]; then + sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ + ${DEBFILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions + fi +} + +addtask cache_config before do_fetch + do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}" # Install Debian packages, that were built from sources -- 2.1.4 ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 2/4] meta-isar-bin: Generate cache repos 2017-10-02 15:45 [PATCH v4 0/4] Basic binary cache implementation Alexander Smirnov 2017-10-02 15:45 ` [PATCH 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov @ 2017-10-02 15:45 ` Alexander Smirnov 2017-10-04 9:03 ` Henning Schild 2017-10-02 15:45 ` [PATCH 3/4] meta-isar-bin: Populate cache Alexander Smirnov ` (2 subsequent siblings) 4 siblings, 1 reply; 19+ messages in thread From: Alexander Smirnov @ 2017-10-02 15:45 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Generate repos for each configuration in multiconfig. Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta-isar-bin/conf/layer.conf | 3 +++ meta-isar/conf/local.conf.sample | 4 ++++ meta/classes/image.bbclass | 35 ++++++++++++++++++++++++++++++++++- 3 files changed, 41 insertions(+), 1 deletion(-) diff --git a/meta-isar-bin/conf/layer.conf b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644 --- a/meta-isar-bin/conf/layer.conf +++ b/meta-isar-bin/conf/layer.conf @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt" # Path to the configuration files templates used by `reprepro` DEBFILESDIR ?= "${LAYERDIR}/files" + +# Path to the databases used by `reprepro` +DEBDBDIR ?= "${LAYERDIR}/db" diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644 --- a/meta-isar/conf/local.conf.sample +++ b/meta-isar/conf/local.conf.sample @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw" # # Default parallel jobs for bitbake: BB_NUMBER_THREADS = "4" + +# +# Number of attempts to try to get reprepro lock for access to apt cache +REPREPRO_LOCK_ATTEMPTS = "16" diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index d30b139..b07775a 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -1,5 +1,5 @@ # This software is a part of ISAR. -# Copyright (C) 2015-2016 ilbers GmbH +# Copyright (C) 2015-2017 ilbers GmbH KERNEL_IMAGE ?= "" INITRD_IMAGE ?= "" @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf" do_cache_config[dirs] = "${CACHE_CONF_DIR}" do_cache_config[stamp-extra-info] = "${DISTRO}" +call_reprepro() { + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do + # According to `sh` manual page, shell exit statuses fall between + # 0-255. The EEXIST error code is (-17), so casting to usigned 8-bit + # integer results value (239). + eexist="239" + retval="0" + reprepro --waitforlock 1 $* || retval="$?" + + # If reprepro has failed to get database lock, it returns EEXIST code. + # In this case we continue trying to get lock until max amount of + # attempts is reached. + if [ $retval -eq $eexist ]; then + bbwarn "Failed to get reprepro lock, trying again..." + else + break + fi + done + + if [ $retval -ne 0 ]; then + bbfatal "reprepro failed" + fi +} + # Generate reprepro config for current distro if it doesn't exist. Once it's # generated, this task should do nothing. do_cache_config() { @@ -21,6 +45,15 @@ do_cache_config() { sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ ${DEBFILESDIR}/distributions.in > ${CACHE_CONF_DIR}/distributions fi + + path_cache="${DEBCACHEDIR}/${DISTRO}" + path_databases="${DEBDBDIR}/${DISTRO}" + + if [ ! -d "${path_databases}" ]; then + call_reprepro -b ${path_cache} \ + --dbdir ${path_databases} \ + export ${DEBDISTRONAME} + fi } addtask cache_config before do_fetch -- 2.1.4 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 2/4] meta-isar-bin: Generate cache repos 2017-10-02 15:45 ` [PATCH 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov @ 2017-10-04 9:03 ` Henning Schild 2017-10-04 10:57 ` Alexander Smirnov 0 siblings, 1 reply; 19+ messages in thread From: Henning Schild @ 2017-10-04 9:03 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users Am Mon, 2 Oct 2017 18:45:29 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > Generate repos for each configuration in multiconfig. > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta-isar-bin/conf/layer.conf | 3 +++ > meta-isar/conf/local.conf.sample | 4 ++++ > meta/classes/image.bbclass | 35 > ++++++++++++++++++++++++++++++++++- 3 files changed, 41 > insertions(+), 1 deletion(-) > > diff --git a/meta-isar-bin/conf/layer.conf > b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644 > --- a/meta-isar-bin/conf/layer.conf > +++ b/meta-isar-bin/conf/layer.conf > @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt" > > # Path to the configuration files templates used by `reprepro` > DEBFILESDIR ?= "${LAYERDIR}/files" > + > +# Path to the databases used by `reprepro` > +DEBDBDIR ?= "${LAYERDIR}/db" > diff --git a/meta-isar/conf/local.conf.sample > b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644 > --- a/meta-isar/conf/local.conf.sample > +++ b/meta-isar/conf/local.conf.sample > @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw" > # > # Default parallel jobs for bitbake: > BB_NUMBER_THREADS = "4" > + > +# > +# Number of attempts to try to get reprepro lock for access to apt > cache +REPREPRO_LOCK_ATTEMPTS = "16" > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index d30b139..b07775a 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -1,5 +1,5 @@ > # This software is a part of ISAR. > -# Copyright (C) 2015-2016 ilbers GmbH > +# Copyright (C) 2015-2017 ilbers GmbH > > KERNEL_IMAGE ?= "" > INITRD_IMAGE ?= "" > @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf" > do_cache_config[dirs] = "${CACHE_CONF_DIR}" > do_cache_config[stamp-extra-info] = "${DISTRO}" > > +call_reprepro() { > + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do > + # According to `sh` manual page, shell exit statuses fall > between > + # 0-255. The EEXIST error code is (-17), so casting to > usigned 8-bit cant you use python to get that number and use the name "EEXIST" instead of a hardcoded number? I think i sent a sniplet in an earlier mail. > + # integer results value (239). > + eexist="239" > + retval="0" > + reprepro --waitforlock 1 $* || retval="$?" The waitforlock should be skipped altogether, which means try it once. The current version means try twice. I got that wrong when i suggested to go for "1", sorry. > + # If reprepro has failed to get database lock, it returns > EEXIST code. > + # In this case we continue trying to get lock until max > amount of > + # attempts is reached. > + if [ $retval -eq $eexist ]; then > + bbwarn "Failed to get reprepro lock, trying again..." Are there other cases where reprepro returns -EEXIST? Henning > + else > + break > + fi > + done > + > + if [ $retval -ne 0 ]; then > + bbfatal "reprepro failed" > + fi > +} > + > # Generate reprepro config for current distro if it doesn't exist. > Once it's # generated, this task should do nothing. > do_cache_config() { > @@ -21,6 +45,15 @@ do_cache_config() { > sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ > ${DEBFILESDIR}/distributions.in > > ${CACHE_CONF_DIR}/distributions fi > + > + path_cache="${DEBCACHEDIR}/${DISTRO}" > + path_databases="${DEBDBDIR}/${DISTRO}" > + > + if [ ! -d "${path_databases}" ]; then > + call_reprepro -b ${path_cache} \ > + --dbdir ${path_databases} \ > + export ${DEBDISTRONAME} > + fi > } > > addtask cache_config before do_fetch ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 2/4] meta-isar-bin: Generate cache repos 2017-10-04 9:03 ` Henning Schild @ 2017-10-04 10:57 ` Alexander Smirnov 2017-10-04 14:09 ` Henning Schild 0 siblings, 1 reply; 19+ messages in thread From: Alexander Smirnov @ 2017-10-04 10:57 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users On 10/04/2017 12:03 PM, Henning Schild wrote: > Am Mon, 2 Oct 2017 18:45:29 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> Generate repos for each configuration in multiconfig. >> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> meta-isar-bin/conf/layer.conf | 3 +++ >> meta-isar/conf/local.conf.sample | 4 ++++ >> meta/classes/image.bbclass | 35 >> ++++++++++++++++++++++++++++++++++- 3 files changed, 41 >> insertions(+), 1 deletion(-) >> >> diff --git a/meta-isar-bin/conf/layer.conf >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644 >> --- a/meta-isar-bin/conf/layer.conf >> +++ b/meta-isar-bin/conf/layer.conf >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt" >> >> # Path to the configuration files templates used by `reprepro` >> DEBFILESDIR ?= "${LAYERDIR}/files" >> + >> +# Path to the databases used by `reprepro` >> +DEBDBDIR ?= "${LAYERDIR}/db" >> diff --git a/meta-isar/conf/local.conf.sample >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644 >> --- a/meta-isar/conf/local.conf.sample >> +++ b/meta-isar/conf/local.conf.sample >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw" >> # >> # Default parallel jobs for bitbake: >> BB_NUMBER_THREADS = "4" >> + >> +# >> +# Number of attempts to try to get reprepro lock for access to apt >> cache +REPREPRO_LOCK_ATTEMPTS = "16" >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index d30b139..b07775a 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -1,5 +1,5 @@ >> # This software is a part of ISAR. >> -# Copyright (C) 2015-2016 ilbers GmbH >> +# Copyright (C) 2015-2017 ilbers GmbH >> >> KERNEL_IMAGE ?= "" >> INITRD_IMAGE ?= "" >> @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf" >> do_cache_config[dirs] = "${CACHE_CONF_DIR}" >> do_cache_config[stamp-extra-info] = "${DISTRO}" >> >> +call_reprepro() { >> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do >> + # According to `sh` manual page, shell exit statuses fall >> between >> + # 0-255. The EEXIST error code is (-17), so casting to >> usigned 8-bit > > cant you use python to get that number and use the name "EEXIST" instead > of a hardcoded number? I think i sent a sniplet in an earlier mail. No we can't. In python I've checked the code EEXIST, and it is (17), while return value from subprocess call is still (239). So using python here is only overhead in correct wrapping of reprepro call. > >> + # integer results value (239). >> + eexist="239" >> + retval="0" >> + reprepro --waitforlock 1 $* || retval="$?" > > The waitforlock should be skipped altogether, which means try it once. > The current version means try twice. I got that wrong when i suggested > to go for "1", sorry. 'waitforlock 1' adds 10 seconds delay between attempts. So the idea to use this form to avoid extra sleep calls with magic timeout value. > >> + # If reprepro has failed to get database lock, it returns >> EEXIST code. >> + # In this case we continue trying to get lock until max >> amount of >> + # attempts is reached. >> + if [ $retval -eq $eexist ]; then >> + bbwarn "Failed to get reprepro lock, trying again..." > > Are there other cases where reprepro returns -EEXIST? > AFAIK there is only one case when reprepro could return EEXIST - it's morgue feature which is not used here. All the other EEXIST checks are processed internally. Alex > Henning > >> + else >> + break >> + fi >> + done >> + >> + if [ $retval -ne 0 ]; then >> + bbfatal "reprepro failed" >> + fi >> +} >> + >> # Generate reprepro config for current distro if it doesn't exist. >> Once it's # generated, this task should do nothing. >> do_cache_config() { >> @@ -21,6 +45,15 @@ do_cache_config() { >> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ >> ${DEBFILESDIR}/distributions.in > >> ${CACHE_CONF_DIR}/distributions fi >> + >> + path_cache="${DEBCACHEDIR}/${DISTRO}" >> + path_databases="${DEBDBDIR}/${DISTRO}" >> + >> + if [ ! -d "${path_databases}" ]; then >> + call_reprepro -b ${path_cache} \ >> + --dbdir ${path_databases} \ >> + export ${DEBDISTRONAME} >> + fi >> } >> >> addtask cache_config before do_fetch > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 2/4] meta-isar-bin: Generate cache repos 2017-10-04 10:57 ` Alexander Smirnov @ 2017-10-04 14:09 ` Henning Schild 0 siblings, 0 replies; 19+ messages in thread From: Henning Schild @ 2017-10-04 14:09 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users Am Wed, 4 Oct 2017 13:57:23 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > On 10/04/2017 12:03 PM, Henning Schild wrote: > > Am Mon, 2 Oct 2017 18:45:29 +0300 > > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > > >> Generate repos for each configuration in multiconfig. > >> > >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > >> --- > >> meta-isar-bin/conf/layer.conf | 3 +++ > >> meta-isar/conf/local.conf.sample | 4 ++++ > >> meta/classes/image.bbclass | 35 > >> ++++++++++++++++++++++++++++++++++- 3 files changed, 41 > >> insertions(+), 1 deletion(-) > >> > >> diff --git a/meta-isar-bin/conf/layer.conf > >> b/meta-isar-bin/conf/layer.conf index 3518184..5ef6e60 100644 > >> --- a/meta-isar-bin/conf/layer.conf > >> +++ b/meta-isar-bin/conf/layer.conf > >> @@ -9,3 +9,6 @@ DEBCACHEDIR ?= "${LAYERDIR}/apt" > >> > >> # Path to the configuration files templates used by `reprepro` > >> DEBFILESDIR ?= "${LAYERDIR}/files" > >> + > >> +# Path to the databases used by `reprepro` > >> +DEBDBDIR ?= "${LAYERDIR}/db" > >> diff --git a/meta-isar/conf/local.conf.sample > >> b/meta-isar/conf/local.conf.sample index a456b1b..660958f 100644 > >> --- a/meta-isar/conf/local.conf.sample > >> +++ b/meta-isar/conf/local.conf.sample > >> @@ -158,3 +158,7 @@ IMAGE_INSTALL = "hello example-raw" > >> # > >> # Default parallel jobs for bitbake: > >> BB_NUMBER_THREADS = "4" > >> + > >> +# > >> +# Number of attempts to try to get reprepro lock for access to apt > >> cache +REPREPRO_LOCK_ATTEMPTS = "16" > >> diff --git a/meta/classes/image.bbclass > >> b/meta/classes/image.bbclass index d30b139..b07775a 100644 > >> --- a/meta/classes/image.bbclass > >> +++ b/meta/classes/image.bbclass > >> @@ -1,5 +1,5 @@ > >> # This software is a part of ISAR. > >> -# Copyright (C) 2015-2016 ilbers GmbH > >> +# Copyright (C) 2015-2017 ilbers GmbH > >> > >> KERNEL_IMAGE ?= "" > >> INITRD_IMAGE ?= "" > >> @@ -14,6 +14,30 @@ CACHE_CONF_DIR = "${DEBCACHEDIR}/${DISTRO}/conf" > >> do_cache_config[dirs] = "${CACHE_CONF_DIR}" > >> do_cache_config[stamp-extra-info] = "${DISTRO}" > >> > >> +call_reprepro() { > >> + for i in $(seq 1 ${REPREPRO_LOCK_ATTEMPTS}); do > >> + # According to `sh` manual page, shell exit statuses fall > >> between > >> + # 0-255. The EEXIST error code is (-17), so casting to > >> usigned 8-bit > > > > cant you use python to get that number and use the name "EEXIST" > > instead of a hardcoded number? I think i sent a sniplet in an > > earlier mail. > > No we can't. In python I've checked the code EEXIST, and it is (17), > while return value from subprocess call is still (239). So using > python here is only overhead in correct wrapping of reprepro call. Just do the math in python, skip the long comment and get rid of the magic number. python -c "import errno; print(256-errno.EEXIST)" > > > >> + # integer results value (239). > >> + eexist="239" > >> + retval="0" > >> + reprepro --waitforlock 1 $* || retval="$?" > > > > The waitforlock should be skipped altogether, which means try it > > once. The current version means try twice. I got that wrong when i > > suggested to go for "1", sorry. > > 'waitforlock 1' adds 10 seconds delay between attempts. So the idea > to use this form to avoid extra sleep calls with magic timeout value. Now you have two nested loops again and the name REPREPRO_LOCK_ATTEMPTS is wrong, because you do it 2*REPREPRO_LOCK_ATTEMPTS. Either divide that number by two or introduce a sleep on the error-path. > > > >> + # If reprepro has failed to get database lock, it returns > >> EEXIST code. > >> + # In this case we continue trying to get lock until max > >> amount of > >> + # attempts is reached. > >> + if [ $retval -eq $eexist ]; then > >> + bbwarn "Failed to get reprepro lock, trying > >> again..." > > > > Are there other cases where reprepro returns -EEXIST? > > > > AFAIK there is only one case when reprepro could return EEXIST - it's > morgue feature which is not used here. All the other EEXIST checks > are processed internally. Ok, just wanted to make sure someone checked that. Henning > Alex > > > Henning > > > >> + else > >> + break > >> + fi > >> + done > >> + > >> + if [ $retval -ne 0 ]; then > >> + bbfatal "reprepro failed" > >> + fi > >> +} > >> + > >> # Generate reprepro config for current distro if it doesn't > >> exist. Once it's # generated, this task should do nothing. > >> do_cache_config() { > >> @@ -21,6 +45,15 @@ do_cache_config() { > >> sed -e "s#{DISTRO_NAME}#"${DEBDISTRONAME}"#g" \ > >> ${DEBFILESDIR}/distributions.in > > >> ${CACHE_CONF_DIR}/distributions fi > >> + > >> + path_cache="${DEBCACHEDIR}/${DISTRO}" > >> + path_databases="${DEBDBDIR}/${DISTRO}" > >> + > >> + if [ ! -d "${path_databases}" ]; then > >> + call_reprepro -b ${path_cache} \ > >> + --dbdir ${path_databases} \ > >> + export ${DEBDISTRONAME} > >> + fi > >> } > >> > >> addtask cache_config before do_fetch > > ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 3/4] meta-isar-bin: Populate cache 2017-10-02 15:45 [PATCH v4 0/4] Basic binary cache implementation Alexander Smirnov 2017-10-02 15:45 ` [PATCH 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov 2017-10-02 15:45 ` [PATCH 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov @ 2017-10-02 15:45 ` Alexander Smirnov 2017-10-02 15:45 ` [PATCH 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov 2017-10-04 8:32 ` [PATCH v4 0/4] Basic binary cache implementation Claudius Heine 4 siblings, 0 replies; 19+ messages in thread From: Alexander Smirnov @ 2017-10-02 15:45 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Add newly built packages to apt cache. Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta/classes/image.bbclass | 8 ++++++++ 1 file changed, 8 insertions(+) diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index b07775a..cf819e6 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -63,6 +63,14 @@ do_populate[stamp-extra-info] = "${DISTRO}-${MACHINE}" # Install Debian packages, that were built from sources do_populate() { if [ -n "${IMAGE_INSTALL}" ]; then + for p in ${IMAGE_INSTALL}; do + call_reprepro -b ${DEBCACHEDIR}/${DISTRO} \ + --dbdir ${DEBDBDIR}/${DISTRO} \ + -C main \ + includedeb ${DEBDISTRONAME} \ + ${DEPLOY_DIR_DEB}/${p}_*.deb + done + sudo mkdir -p ${IMAGE_ROOTFS}/deb for p in ${IMAGE_INSTALL}; do -- 2.1.4 ^ permalink raw reply [flat|nested] 19+ messages in thread
* [PATCH 4/4] meta-isar-bin: Install packages via multistrap 2017-10-02 15:45 [PATCH v4 0/4] Basic binary cache implementation Alexander Smirnov ` (2 preceding siblings ...) 2017-10-02 15:45 ` [PATCH 3/4] meta-isar-bin: Populate cache Alexander Smirnov @ 2017-10-02 15:45 ` Alexander Smirnov 2017-10-04 9:05 ` Henning Schild 2017-10-04 8:32 ` [PATCH v4 0/4] Basic binary cache implementation Claudius Heine 4 siblings, 1 reply; 19+ messages in thread From: Alexander Smirnov @ 2017-10-02 15:45 UTC (permalink / raw) To: isar-users; +Cc: Alexander Smirnov Install custom packages to target image filesystem using multistrap. Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> --- meta-isar/recipes-core/images/files/multistrap.conf.in | 9 ++++++++- meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++- meta/classes/ext4-img.bbclass | 2 +- meta/classes/image.bbclass | 10 ---------- 4 files changed, 13 insertions(+), 13 deletions(-) diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in b/meta-isar/recipes-core/images/files/multistrap.conf.in index 2d08c36..38d53cf 100644 --- a/meta-isar/recipes-core/images/files/multistrap.conf.in +++ b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7 @@ [General] noauth=true unpack=true -bootstrap=##DISTRO## +bootstrap=##DISTRO## Isar aptsources=##DISTRO## configscript=##CONFIG_SCRIPT## setupscript=##SETUP_SCRIPT## @@ -16,3 +16,10 @@ suite=##DISTRO_SUITE## components=##DISTRO_COMPONENTS## packages=##IMAGE_PREINSTALL## omitdebsrc=true + +[Isar] +packages=##IMAGE_INSTALL## +source=##DEPLOY_DIR_DEB## +suite=##ISAR_DISTRO_SUITE## +components=main +omitdebsrc=true diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb index f4b0125..3b29a53 100644 --- a/meta-isar/recipes-core/images/isar-image-base.bb +++ b/meta-isar/recipes-core/images/isar-image-base.bb @@ -46,6 +46,9 @@ do_rootfs() { -e 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ -e 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ + -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ + -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' \ + -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ "${WORKDIR}/multistrap.conf.in" > "${WORKDIR}/multistrap.conf" # Create root filesystem @@ -57,4 +60,4 @@ do_rootfs() { sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}" } -addtask rootfs before do_populate after do_unpack +addtask rootfs before do_build after do_populate diff --git a/meta/classes/ext4-img.bbclass b/meta/classes/ext4-img.bbclass index 3e3768b..9fa9aa5 100644 --- a/meta/classes/ext4-img.bbclass +++ b/meta/classes/ext4-img.bbclass @@ -34,4 +34,4 @@ do_ext4_image() { fi } -addtask ext4_image before do_build after do_populate +addtask ext4_image before do_build after do_rootfs diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass index cf819e6..ff8a370 100644 --- a/meta/classes/image.bbclass +++ b/meta/classes/image.bbclass @@ -70,16 +70,6 @@ do_populate() { includedeb ${DEBDISTRONAME} \ ${DEPLOY_DIR_DEB}/${p}_*.deb done - - sudo mkdir -p ${IMAGE_ROOTFS}/deb - - for p in ${IMAGE_INSTALL}; do - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb - done - - sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb - - sudo rm -rf ${IMAGE_ROOTFS}/deb fi } -- 2.1.4 ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 4/4] meta-isar-bin: Install packages via multistrap 2017-10-02 15:45 ` [PATCH 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov @ 2017-10-04 9:05 ` Henning Schild 2017-10-04 10:59 ` Alexander Smirnov 0 siblings, 1 reply; 19+ messages in thread From: Henning Schild @ 2017-10-04 9:05 UTC (permalink / raw) To: Alexander Smirnov; +Cc: isar-users Am Mon, 2 Oct 2017 18:45:31 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > Install custom packages to target image filesystem using multistrap. > > Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> > --- > meta-isar/recipes-core/images/files/multistrap.conf.in | 9 ++++++++- > meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++- > meta/classes/ext4-img.bbclass | 2 +- > meta/classes/image.bbclass | 10 > ---------- 4 files changed, 13 insertions(+), 13 deletions(-) > > diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in > b/meta-isar/recipes-core/images/files/multistrap.conf.in index > 2d08c36..38d53cf 100644 --- > a/meta-isar/recipes-core/images/files/multistrap.conf.in +++ > b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7 > @@ [General] > noauth=true > unpack=true > -bootstrap=##DISTRO## > +bootstrap=##DISTRO## Isar > aptsources=##DISTRO## > configscript=##CONFIG_SCRIPT## > setupscript=##SETUP_SCRIPT## > @@ -16,3 +16,10 @@ suite=##DISTRO_SUITE## > components=##DISTRO_COMPONENTS## > packages=##IMAGE_PREINSTALL## > omitdebsrc=true > + > +[Isar] > +packages=##IMAGE_INSTALL## > +source=##DEPLOY_DIR_DEB## > +suite=##ISAR_DISTRO_SUITE## > +components=main > +omitdebsrc=true > diff --git a/meta-isar/recipes-core/images/isar-image-base.bb > b/meta-isar/recipes-core/images/isar-image-base.bb index > f4b0125..3b29a53 100644 --- > a/meta-isar/recipes-core/images/isar-image-base.bb +++ > b/meta-isar/recipes-core/images/isar-image-base.bb @@ -46,6 +46,9 @@ > do_rootfs() { -e > 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ > -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ -e > 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ > + -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ > + -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' > \ > + -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ > "${WORKDIR}/multistrap.conf.in" > > "${WORKDIR}/multistrap.conf" > # Create root filesystem > @@ -57,4 +60,4 @@ do_rootfs() { > sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}" > } > > -addtask rootfs before do_populate after do_unpack > +addtask rootfs before do_build after do_populate > diff --git a/meta/classes/ext4-img.bbclass > b/meta/classes/ext4-img.bbclass index 3e3768b..9fa9aa5 100644 > --- a/meta/classes/ext4-img.bbclass > +++ b/meta/classes/ext4-img.bbclass > @@ -34,4 +34,4 @@ do_ext4_image() { > fi > } > > -addtask ext4_image before do_build after do_populate > +addtask ext4_image before do_build after do_rootfs > diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass > index cf819e6..ff8a370 100644 > --- a/meta/classes/image.bbclass > +++ b/meta/classes/image.bbclass > @@ -70,16 +70,6 @@ do_populate() { > includedeb ${DEBDISTRONAME} \ > ${DEPLOY_DIR_DEB}/${p}_*.deb > done > - > - sudo mkdir -p ${IMAGE_ROOTFS}/deb > - > - for p in ${IMAGE_INSTALL}; do > - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb > - done > - > - sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb > - > - sudo rm -rf ${IMAGE_ROOTFS}/deb The comment of what do_populate does is wrong after that. Henning > fi > } > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH 4/4] meta-isar-bin: Install packages via multistrap 2017-10-04 9:05 ` Henning Schild @ 2017-10-04 10:59 ` Alexander Smirnov 0 siblings, 0 replies; 19+ messages in thread From: Alexander Smirnov @ 2017-10-04 10:59 UTC (permalink / raw) To: Henning Schild; +Cc: isar-users On 10/04/2017 12:05 PM, Henning Schild wrote: > Am Mon, 2 Oct 2017 18:45:31 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> Install custom packages to target image filesystem using multistrap. >> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de> >> --- >> meta-isar/recipes-core/images/files/multistrap.conf.in | 9 ++++++++- >> meta-isar/recipes-core/images/isar-image-base.bb | 5 ++++- >> meta/classes/ext4-img.bbclass | 2 +- >> meta/classes/image.bbclass | 10 >> ---------- 4 files changed, 13 insertions(+), 13 deletions(-) >> >> diff --git a/meta-isar/recipes-core/images/files/multistrap.conf.in >> b/meta-isar/recipes-core/images/files/multistrap.conf.in index >> 2d08c36..38d53cf 100644 --- >> a/meta-isar/recipes-core/images/files/multistrap.conf.in +++ >> b/meta-isar/recipes-core/images/files/multistrap.conf.in @@ -4,7 +4,7 >> @@ [General] >> noauth=true >> unpack=true >> -bootstrap=##DISTRO## >> +bootstrap=##DISTRO## Isar >> aptsources=##DISTRO## >> configscript=##CONFIG_SCRIPT## >> setupscript=##SETUP_SCRIPT## >> @@ -16,3 +16,10 @@ suite=##DISTRO_SUITE## >> components=##DISTRO_COMPONENTS## >> packages=##IMAGE_PREINSTALL## >> omitdebsrc=true >> + >> +[Isar] >> +packages=##IMAGE_INSTALL## >> +source=##DEPLOY_DIR_DEB## >> +suite=##ISAR_DISTRO_SUITE## >> +components=main >> +omitdebsrc=true >> diff --git a/meta-isar/recipes-core/images/isar-image-base.bb >> b/meta-isar/recipes-core/images/isar-image-base.bb index >> f4b0125..3b29a53 100644 --- >> a/meta-isar/recipes-core/images/isar-image-base.bb +++ >> b/meta-isar/recipes-core/images/isar-image-base.bb @@ -46,6 +46,9 @@ >> do_rootfs() { -e >> 's|##CONFIG_SCRIPT##|./'"$WORKDIR_REL"'/${DISTRO_CONFIG_SCRIPT}|g' \ >> -e 's|##SETUP_SCRIPT##|./'"$WORKDIR_REL"'/setup.sh|g' \ -e >> 's|##DIR_HOOKS##|./'"$WORKDIR_REL"'/hooks_multistrap|g' \ >> + -e 's|##IMAGE_INSTALL##|${IMAGE_INSTALL}|g' \ >> + -e 's|##DEPLOY_DIR_DEB##|copy:///${DEBCACHEDIR}/${DISTRO}|g' >> \ >> + -e 's|##ISAR_DISTRO_SUITE##|${DEBDISTRONAME}|g' \ >> "${WORKDIR}/multistrap.conf.in" > >> "${WORKDIR}/multistrap.conf" >> # Create root filesystem >> @@ -57,4 +60,4 @@ do_rootfs() { >> sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}" >> } >> >> -addtask rootfs before do_populate after do_unpack >> +addtask rootfs before do_build after do_populate >> diff --git a/meta/classes/ext4-img.bbclass >> b/meta/classes/ext4-img.bbclass index 3e3768b..9fa9aa5 100644 >> --- a/meta/classes/ext4-img.bbclass >> +++ b/meta/classes/ext4-img.bbclass >> @@ -34,4 +34,4 @@ do_ext4_image() { >> fi >> } >> >> -addtask ext4_image before do_build after do_populate >> +addtask ext4_image before do_build after do_rootfs >> diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass >> index cf819e6..ff8a370 100644 >> --- a/meta/classes/image.bbclass >> +++ b/meta/classes/image.bbclass >> @@ -70,16 +70,6 @@ do_populate() { >> includedeb ${DEBDISTRONAME} \ >> ${DEPLOY_DIR_DEB}/${p}_*.deb >> done >> - >> - sudo mkdir -p ${IMAGE_ROOTFS}/deb >> - >> - for p in ${IMAGE_INSTALL}; do >> - sudo cp ${DEPLOY_DIR_DEB}/${p}_*.deb ${IMAGE_ROOTFS}/deb >> - done >> - >> - sudo chroot ${IMAGE_ROOTFS} /usr/bin/dpkg -i -R /deb >> - >> - sudo rm -rf ${IMAGE_ROOTFS}/deb > > The comment of what do_populate does is wrong after that. Thanks for this hint. > >> fi >> } >> ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-02 15:45 [PATCH v4 0/4] Basic binary cache implementation Alexander Smirnov ` (3 preceding siblings ...) 2017-10-02 15:45 ` [PATCH 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov @ 2017-10-04 8:32 ` Claudius Heine 2017-10-04 11:57 ` Jan Kiszka 4 siblings, 1 reply; 19+ messages in thread From: Claudius Heine @ 2017-10-04 8:32 UTC (permalink / raw) To: Alexander Smirnov, isar-users Hi, On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: > Hello everybody, > > this series introduces basic binary caching for Isar. > > There is a new layer: meta-isar-bin which is intended to be the binary > cache. All the packages that are built in Isar are stored in this > cache using reprepro. > > Having the separate layer makes possible to manage this cache separately > from the Isar build tree. So once you have built your packages, you could > re-use this cache for further builds. We still have to talk about your requirement for the cache to be sharable and your resulting implementation of it as a meta layer. Since I still don't see the benefit of a sharable cache and think that putting binary packages into some kind of strange meta layer is not the right solution and could potentially create more problems that is solves because its very much in conflict with the fundamental idea of a meta layer. Please give me a reason for this requirement. And to be precise: Yes, a cache is useful and I fully support it, but having it sharable over multiple hosts is not and is potentially dangerous. Claudius > > Also this cache repo is now used to generate image filesystem via > multistrap. > > Changes since v3: > - Check return code from reprepro, continue loop only if EEXIST code is > returned. Othrwise build is aborted. > > Changes since v2: > - Droped anonymous functions > - Added number of attempts to access apt cache before fail > > Changes since v1: > - Total refactoring > - Split huge patches into small pieces > - Avoid races > > With best regards, > Alex > > Alexander Smirnov (4): > meta-isar-bin: Add reprepro configs > meta-isar-bin: Generate cache repos > meta-isar-bin: Populate cache > meta-isar-bin: Install packages via multistrap > > meta-isar-bin/conf/layer.conf | 14 +++++ > meta-isar-bin/files/distributions.in | 3 ++ > meta-isar/conf/bblayers.conf.sample | 1 + > meta-isar/conf/local.conf.sample | 4 ++ > .../recipes-core/images/files/multistrap.conf.in | 9 +++- > meta-isar/recipes-core/images/isar-image-base.bb | 5 +- > meta/classes/ext4-img.bbclass | 2 +- > meta/classes/image.bbclass | 62 +++++++++++++++++++--- > 8 files changed, 89 insertions(+), 11 deletions(-) > create mode 100644 meta-isar-bin/conf/layer.conf > create mode 100644 meta-isar-bin/files/distributions.in > -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 8:32 ` [PATCH v4 0/4] Basic binary cache implementation Claudius Heine @ 2017-10-04 11:57 ` Jan Kiszka 2017-10-04 14:29 ` Alexander Smirnov 0 siblings, 1 reply; 19+ messages in thread From: Jan Kiszka @ 2017-10-04 11:57 UTC (permalink / raw) To: [ext] Claudius Heine, Alexander Smirnov, isar-users On 2017-10-04 10:32, [ext] Claudius Heine wrote: > Hi, > > On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: >> Hello everybody, >> >> this series introduces basic binary caching for Isar. >> >> There is a new layer: meta-isar-bin which is intended to be the binary >> cache. All the packages that are built in Isar are stored in this >> cache using reprepro. >> >> Having the separate layer makes possible to manage this cache separately >> from the Isar build tree. So once you have built your packages, you could >> re-use this cache for further builds. > > We still have to talk about your requirement for the cache to be > sharable and your resulting implementation of it as a meta layer. > > Since I still don't see the benefit of a sharable cache and think that > putting binary packages into some kind of strange meta layer is not the > right solution and could potentially create more problems that is solves > because its very much in conflict with the fundamental idea of a meta > layer. Indeed, state like this (disclaimer: i didn't look into all details), it sounds like a weird architecture to me as well. Thinking of an artifact cache, I would rather expect something like OE's sstate that one may preserve across builds, share between devs, or whatever. If it's there, the build system consults it, if not, it rebuilds. But sstate is not a layer, for some good reasons. Jan -- Siemens AG, Corporate Technology, CT RDA ITP SES-DE Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 11:57 ` Jan Kiszka @ 2017-10-04 14:29 ` Alexander Smirnov 2017-10-04 15:40 ` Henning Schild 2017-10-04 17:34 ` Claudius Heine 0 siblings, 2 replies; 19+ messages in thread From: Alexander Smirnov @ 2017-10-04 14:29 UTC (permalink / raw) To: Jan Kiszka, [ext] Claudius Heine, isar-users On 10/04/2017 02:57 PM, Jan Kiszka wrote: > On 2017-10-04 10:32, [ext] Claudius Heine wrote: >> Hi, >> >> On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: >>> Hello everybody, >>> >>> this series introduces basic binary caching for Isar. >>> >>> There is a new layer: meta-isar-bin which is intended to be the binary >>> cache. All the packages that are built in Isar are stored in this >>> cache using reprepro. >>> >>> Having the separate layer makes possible to manage this cache separately >>> from the Isar build tree. So once you have built your packages, you could >>> re-use this cache for further builds. >> >> We still have to talk about your requirement for the cache to be >> sharable and your resulting implementation of it as a meta layer. >> >> Since I still don't see the benefit of a sharable cache and think that >> putting binary packages into some kind of strange meta layer is not the >> right solution and could potentially create more problems that is solves >> because its very much in conflict with the fundamental idea of a meta >> layer. The last statement is a bit strange for me. The idea of meta layers is to split software stack into logical layers that could be enabled/disabled on demand. Having binaries or recipes doesn't matter here, it's just the way how software is provided to build. > > Indeed, state like this (disclaimer: i didn't look into all details), it > sounds like a weird architecture to me as well. Thinking of an artifact > cache, I would rather expect something like OE's sstate that one may > preserve across builds, share between devs, or whatever. If it's there, > the build system consults it, if not, it rebuilds. But sstate is not a > layer, for some good reasons. > Let me again summarize major points here. Isar is the build system designed to work with binary packages. That's the key feature of this product and main difference from OE-like systems. The Isar architecture assumes to be designed around binary packages, what provides absolutely different approach how binary images could be generated using bitbake. According to the Claudius's vision, that everything should always be built from sources - Isar is not the best option here, Yocto and OE are much better, because they were initially designed for this work. Attempts to apply OE design and philosophy to Isar could have negative impact and limit possible ways to implement custom features, because OE wasn't designed for this features. Also I wonder why we are able to use binary Debian packages, while the rest of software should be built from sources. This series introduces support for binary apt repositories in Isar. This feature is response to the following cases: 1. Publish custom apt layer in web. When I need to install some custom software to my Debian system, I frequently see instructions like: - Add 'blablabla' to your /etc/apt/source.list - Run apt-get update - Run apt-get install That's for example how Jenkins and GitLab are distributed: https://pkg.jenkins.io/debian/ The management of this public apt could require some manual effort, while Isar (in plans) could update it automatically, so you update source code for your product, run Isar and it provides complete apt that you could upload to your server. 2. Implement easy software update. On our host systems we usually do not flash whole image if we need to update some packages. The update is performed using 'apt-get install' (or by similar tool). But in this case binary repo is needed, what could be again provided by Isar. Please do not limit Isar by deep embedded application, host systems also could be managed by Isar. 3. In long term, help Debian community to manage Debian infrastructure, what includes publishing apt updates. -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 14:29 ` Alexander Smirnov @ 2017-10-04 15:40 ` Henning Schild 2017-10-04 16:50 ` Alexander Smirnov 2017-10-04 17:34 ` Claudius Heine 1 sibling, 1 reply; 19+ messages in thread From: Henning Schild @ 2017-10-04 15:40 UTC (permalink / raw) To: Alexander Smirnov; +Cc: Jan Kiszka, [ext] Claudius Heine, isar-users Am Wed, 4 Oct 2017 17:29:08 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > On 10/04/2017 02:57 PM, Jan Kiszka wrote: > > On 2017-10-04 10:32, [ext] Claudius Heine wrote: > >> Hi, > >> > >> On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: > >>> Hello everybody, > >>> > >>> this series introduces basic binary caching for Isar. > >>> > >>> There is a new layer: meta-isar-bin which is intended to be the > >>> binary cache. All the packages that are built in Isar are stored > >>> in this cache using reprepro. > >>> > >>> Having the separate layer makes possible to manage this cache > >>> separately from the Isar build tree. So once you have built your > >>> packages, you could re-use this cache for further builds. > >> > >> We still have to talk about your requirement for the cache to be > >> sharable and your resulting implementation of it as a meta layer. > >> > >> Since I still don't see the benefit of a sharable cache and think > >> that putting binary packages into some kind of strange meta layer > >> is not the right solution and could potentially create more > >> problems that is solves because its very much in conflict with the > >> fundamental idea of a meta layer. > > The last statement is a bit strange for me. The idea of meta layers > is to split software stack into logical layers that could be > enabled/disabled on demand. Having binaries or recipes doesn't matter > here, it's just the way how software is provided to build. Binaries that can be rebuild should not become part of the build system. The need to distribute them to the targets or other developers/images later is what we all agree on. But that should be mostly outside of isar. Building an Image will result in the Image and a repo. Both have to somehow make their way to the target, but that is somewhat out of scope of the build system. > > > > Indeed, state like this (disclaimer: i didn't look into all > > details), it sounds like a weird architecture to me as well. > > Thinking of an artifact cache, I would rather expect something like > > OE's sstate that one may preserve across builds, share between > > devs, or whatever. If it's there, the build system consults it, if > > not, it rebuilds. But sstate is not a layer, for some good reasons. > > > > Let me again summarize major points here. Isar is the build system > designed to work with binary packages. That's the key feature of this > product and main difference from OE-like systems. The Isar > architecture assumes to be designed around binary packages, what > provides absolutely different approach how binary images could be > generated using bitbake. > > According to the Claudius's vision, that everything should always be > built from sources - Isar is not the best option here, Yocto and OE > are much better, because they were initially designed for this work. > Attempts to apply OE design and philosophy to Isar could have > negative impact and limit possible ways to implement custom features, > because OE wasn't designed for this features. > Also I wonder why we are able to use binary Debian packages, while > the rest of software should be built from sources. > > This series introduces support for binary apt repositories in Isar. > This feature is response to the following cases: > > 1. Publish custom apt layer in web. When I need to install some > custom software to my Debian system, I frequently see instructions > like: > - Add 'blablabla' to your /etc/apt/source.list > - Run apt-get update > - Run apt-get install > > That's for example how Jenkins and GitLab are distributed: > > https://pkg.jenkins.io/debian/ I think we all get that and agree on it. > The management of this public apt could require some manual effort, > while Isar (in plans) could update it automatically, so you update > source code for your product, run Isar and it provides complete apt > that you could upload to your server. Exactly, those manual steps or custom scripts are outside the scope of Isar. Just like the flashing of the Images onto the targets. > 2. Implement easy software update. On our host systems we usually do > not flash whole image if we need to update some packages. The update > is performed using 'apt-get install' (or by similar tool). But in > this case binary repo is needed, what could be again provided by Isar. > > Please do not limit Isar by deep embedded application, host systems > also could be managed by Isar. If the repo ends up in the output directory all those scenarios are still possible, _and_ your source tree will not contain build-output. Henning > 3. In long term, help Debian community to manage Debian > infrastructure, what includes publishing apt updates. > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 15:40 ` Henning Schild @ 2017-10-04 16:50 ` Alexander Smirnov 2017-10-04 16:58 ` Henning Schild 0 siblings, 1 reply; 19+ messages in thread From: Alexander Smirnov @ 2017-10-04 16:50 UTC (permalink / raw) To: Henning Schild; +Cc: Jan Kiszka, [ext] Claudius Heine, isar-users Hi, On 10/04/2017 06:40 PM, Henning Schild wrote: > Am Wed, 4 Oct 2017 17:29:08 +0300 > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > >> On 10/04/2017 02:57 PM, Jan Kiszka wrote: >>> On 2017-10-04 10:32, [ext] Claudius Heine wrote: >>>> Hi, >>>> >>>> On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: >>>>> Hello everybody, >>>>> >>>>> this series introduces basic binary caching for Isar. >>>>> >>>>> There is a new layer: meta-isar-bin which is intended to be the >>>>> binary cache. All the packages that are built in Isar are stored >>>>> in this cache using reprepro. >>>>> >>>>> Having the separate layer makes possible to manage this cache >>>>> separately from the Isar build tree. So once you have built your >>>>> packages, you could re-use this cache for further builds. >>>> >>>> We still have to talk about your requirement for the cache to be >>>> sharable and your resulting implementation of it as a meta layer. >>>> >>>> Since I still don't see the benefit of a sharable cache and think >>>> that putting binary packages into some kind of strange meta layer >>>> is not the right solution and could potentially create more >>>> problems that is solves because its very much in conflict with the >>>> fundamental idea of a meta layer. >> >> The last statement is a bit strange for me. The idea of meta layers >> is to split software stack into logical layers that could be >> enabled/disabled on demand. Having binaries or recipes doesn't matter >> here, it's just the way how software is provided to build. > > Binaries that can be rebuild should not become part of the build > system. The need to distribute them to the targets or other > developers/images later is what we all agree on. But that should be > mostly outside of isar. Each binary could be re-built from sources (at least for Debian :-) ). Moreover Debian provides sources for them, so they are open for use. But for me it looks like: - Installing binaries to target from 'ftp.debian.org' is allowed. - Installing binary dependencies for custom built package from 'ftp.debian.org' is allowed. - Installing my binary application from 'my.debian.repo' is not allowed. I have to build it from sources. What is the idea here? For me it's something like traditional OE philosophy, but without benefit for Isar. One of the typical usecase: if I'm, for example, developing 'hello' application, why it's so critical for me to build, lets say, kernel from sources? Alex > > Building an Image will result in the Image and a repo. Both have to > somehow make their way to the target, but that is somewhat out of scope > of the build system. > >>> >>> Indeed, state like this (disclaimer: i didn't look into all >>> details), it sounds like a weird architecture to me as well. >>> Thinking of an artifact cache, I would rather expect something like >>> OE's sstate that one may preserve across builds, share between >>> devs, or whatever. If it's there, the build system consults it, if >>> not, it rebuilds. But sstate is not a layer, for some good reasons. >>> >> >> Let me again summarize major points here. Isar is the build system >> designed to work with binary packages. That's the key feature of this >> product and main difference from OE-like systems. The Isar >> architecture assumes to be designed around binary packages, what >> provides absolutely different approach how binary images could be >> generated using bitbake. >> >> According to the Claudius's vision, that everything should always be >> built from sources - Isar is not the best option here, Yocto and OE >> are much better, because they were initially designed for this work. >> Attempts to apply OE design and philosophy to Isar could have >> negative impact and limit possible ways to implement custom features, >> because OE wasn't designed for this features. >> Also I wonder why we are able to use binary Debian packages, while >> the rest of software should be built from sources. >> >> This series introduces support for binary apt repositories in Isar. >> This feature is response to the following cases: >> >> 1. Publish custom apt layer in web. When I need to install some >> custom software to my Debian system, I frequently see instructions >> like: >> - Add 'blablabla' to your /etc/apt/source.list >> - Run apt-get update >> - Run apt-get install >> >> That's for example how Jenkins and GitLab are distributed: >> >> https://pkg.jenkins.io/debian/ > > I think we all get that and agree on it. > >> The management of this public apt could require some manual effort, >> while Isar (in plans) could update it automatically, so you update >> source code for your product, run Isar and it provides complete apt >> that you could upload to your server. > > Exactly, those manual steps or custom scripts are outside the scope of > Isar. Just like the flashing of the Images onto the targets. > >> 2. Implement easy software update. On our host systems we usually do >> not flash whole image if we need to update some packages. The update >> is performed using 'apt-get install' (or by similar tool). But in >> this case binary repo is needed, what could be again provided by Isar. >> >> Please do not limit Isar by deep embedded application, host systems >> also could be managed by Isar. > > If the repo ends up in the output directory all those scenarios are > still possible, _and_ your source tree will not contain build-output. > > Henning > >> 3. In long term, help Debian community to manage Debian >> infrastructure, what includes publishing apt updates. >> > -- With best regards, Alexander Smirnov ilbers GmbH Baierbrunner Str. 28c D-81379 Munich +49 (89) 122 67 24-0 http://ilbers.de/ Commercial register Munich, HRB 214197 General manager: Baurzhan Ismagulov ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 16:50 ` Alexander Smirnov @ 2017-10-04 16:58 ` Henning Schild 0 siblings, 0 replies; 19+ messages in thread From: Henning Schild @ 2017-10-04 16:58 UTC (permalink / raw) To: Alexander Smirnov; +Cc: Jan Kiszka, [ext] Claudius Heine, isar-users Am Wed, 4 Oct 2017 19:50:09 +0300 schrieb Alexander Smirnov <asmirnov@ilbers.de>: > Hi, > > On 10/04/2017 06:40 PM, Henning Schild wrote: > > Am Wed, 4 Oct 2017 17:29:08 +0300 > > schrieb Alexander Smirnov <asmirnov@ilbers.de>: > > > >> On 10/04/2017 02:57 PM, Jan Kiszka wrote: > >>> On 2017-10-04 10:32, [ext] Claudius Heine wrote: > >>>> Hi, > >>>> > >>>> On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: > >>>>> Hello everybody, > >>>>> > >>>>> this series introduces basic binary caching for Isar. > >>>>> > >>>>> There is a new layer: meta-isar-bin which is intended to be the > >>>>> binary cache. All the packages that are built in Isar are stored > >>>>> in this cache using reprepro. > >>>>> > >>>>> Having the separate layer makes possible to manage this cache > >>>>> separately from the Isar build tree. So once you have built your > >>>>> packages, you could re-use this cache for further builds. > >>>> > >>>> We still have to talk about your requirement for the cache to be > >>>> sharable and your resulting implementation of it as a meta layer. > >>>> > >>>> Since I still don't see the benefit of a sharable cache and think > >>>> that putting binary packages into some kind of strange meta layer > >>>> is not the right solution and could potentially create more > >>>> problems that is solves because its very much in conflict with > >>>> the fundamental idea of a meta layer. > >> > >> The last statement is a bit strange for me. The idea of meta layers > >> is to split software stack into logical layers that could be > >> enabled/disabled on demand. Having binaries or recipes doesn't > >> matter here, it's just the way how software is provided to build. > > > > Binaries that can be rebuild should not become part of the build > > system. The need to distribute them to the targets or other > > developers/images later is what we all agree on. But that should be > > mostly outside of isar. > > Each binary could be re-built from sources (at least for Debian :-) > ). Moreover Debian provides sources for them, so they are open for > use. > > But for me it looks like: > > - Installing binaries to target from 'ftp.debian.org' is allowed. > - Installing binary dependencies for custom built package from > 'ftp.debian.org' is allowed. > - Installing my binary application from 'my.debian.repo' is not > allowed. I have to build it from sources. That is not only allowed it is absolutely wanted! But for that step you would add that repo to your multistrap config or /etc/apt/, which is in the scope of isar. Building the repo is also in the scope of isar, but holding the artefacts and publishing the repo outside the build-dir is not. Henning > What is the idea here? For me it's something like traditional OE > philosophy, but without benefit for Isar. > > One of the typical usecase: if I'm, for example, developing 'hello' > application, why it's so critical for me to build, lets say, kernel > from sources? > > Alex > > > > > Building an Image will result in the Image and a repo. Both have to > > somehow make their way to the target, but that is somewhat out of > > scope of the build system. > > > >>> > >>> Indeed, state like this (disclaimer: i didn't look into all > >>> details), it sounds like a weird architecture to me as well. > >>> Thinking of an artifact cache, I would rather expect something > >>> like OE's sstate that one may preserve across builds, share > >>> between devs, or whatever. If it's there, the build system > >>> consults it, if not, it rebuilds. But sstate is not a layer, for > >>> some good reasons. > >> > >> Let me again summarize major points here. Isar is the build system > >> designed to work with binary packages. That's the key feature of > >> this product and main difference from OE-like systems. The Isar > >> architecture assumes to be designed around binary packages, what > >> provides absolutely different approach how binary images could be > >> generated using bitbake. > >> > >> According to the Claudius's vision, that everything should always > >> be built from sources - Isar is not the best option here, Yocto > >> and OE are much better, because they were initially designed for > >> this work. Attempts to apply OE design and philosophy to Isar > >> could have negative impact and limit possible ways to implement > >> custom features, because OE wasn't designed for this features. > >> Also I wonder why we are able to use binary Debian packages, while > >> the rest of software should be built from sources. > >> > >> This series introduces support for binary apt repositories in Isar. > >> This feature is response to the following cases: > >> > >> 1. Publish custom apt layer in web. When I need to install some > >> custom software to my Debian system, I frequently see instructions > >> like: > >> - Add 'blablabla' to your /etc/apt/source.list > >> - Run apt-get update > >> - Run apt-get install > >> > >> That's for example how Jenkins and GitLab are distributed: > >> > >> https://pkg.jenkins.io/debian/ > > > > I think we all get that and agree on it. > > > >> The management of this public apt could require some manual effort, > >> while Isar (in plans) could update it automatically, so you update > >> source code for your product, run Isar and it provides complete apt > >> that you could upload to your server. > > > > Exactly, those manual steps or custom scripts are outside the scope > > of Isar. Just like the flashing of the Images onto the targets. > > > >> 2. Implement easy software update. On our host systems we usually > >> do not flash whole image if we need to update some packages. The > >> update is performed using 'apt-get install' (or by similar tool). > >> But in this case binary repo is needed, what could be again > >> provided by Isar. > >> > >> Please do not limit Isar by deep embedded application, host systems > >> also could be managed by Isar. > > > > If the repo ends up in the output directory all those scenarios are > > still possible, _and_ your source tree will not contain > > build-output. > > Henning > > > >> 3. In long term, help Debian community to manage Debian > >> infrastructure, what includes publishing apt updates. > >> > > > ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 14:29 ` Alexander Smirnov 2017-10-04 15:40 ` Henning Schild @ 2017-10-04 17:34 ` Claudius Heine 2017-10-04 18:47 ` Alexander Smirnov 1 sibling, 1 reply; 19+ messages in thread From: Claudius Heine @ 2017-10-04 17:34 UTC (permalink / raw) To: Alexander Smirnov, Jan Kiszka, [ext] Claudius Heine, isar-users [-- Attachment #1: Type: text/plain, Size: 4820 bytes --] Hi Alex, On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote: > On 10/04/2017 02:57 PM, Jan Kiszka wrote: > > On 2017-10-04 10:32, [ext] Claudius Heine wrote: > > > Hi, > > > > > > On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: > > > > Hello everybody, > > > > > > > > this series introduces basic binary caching for Isar. > > > > > > > > There is a new layer: meta-isar-bin which is intended to be the > > > > binary > > > > cache. All the packages that are built in Isar are stored in > > > > this > > > > cache using reprepro. > > > > > > > > Having the separate layer makes possible to manage this cache > > > > separately > > > > from the Isar build tree. So once you have built your packages, > > > > you could > > > > re-use this cache for further builds. > > > > > > We still have to talk about your requirement for the cache to be > > > sharable and your resulting implementation of it as a meta layer. > > > > > > Since I still don't see the benefit of a sharable cache and think > > > that > > > putting binary packages into some kind of strange meta layer is > > > not the > > > right solution and could potentially create more problems that is > > > solves > > > because its very much in conflict with the fundamental idea of a > > > meta > > > layer. > > The last statement is a bit strange for me. The idea of meta layers > is > to split software stack into logical layers that could be > enabled/disabled on demand. Having binaries or recipes doesn't > matter > here, it's just the way how software is provided to build. > > > > > Indeed, state like this (disclaimer: i didn't look into all > > details), it > > sounds like a weird architecture to me as well. Thinking of an > > artifact > > cache, I would rather expect something like OE's sstate that one > > may > > preserve across builds, share between devs, or whatever. If it's > > there, > > the build system consults it, if not, it rebuilds. But sstate is > > not a > > layer, for some good reasons. > > > > Let me again summarize major points here. Isar is the build system > designed to work with binary packages. That's the key feature of > this > product and main difference from OE-like systems. The Isar > architecture > assumes to be designed around binary packages, what provides > absolutely > different approach how binary images could be generated using > bitbake. > > According to the Claudius's vision, that everything should always be > built from sources - Isar is not the best option here, Yocto and OE > are > much better, because they were initially designed for this work. > Attempts to apply OE design and philosophy to Isar could have > negative > impact and limit possible ways to implement custom features, because > OE > wasn't designed for this features. > Also I wonder why we are able to use binary Debian packages, while > the > rest of software should be built from sources. You are still not understanding me. I never said you have to build every package from source code in isar. Instead I mean that you have to build the root filesystem / images / packages ... ( every output of isar) from all the available input of isar (upstream debian repositories, meta-layers, source code repositories, etc.). If you create intermediate products in the form of caches, then that is ok. But if you put those intermediate products into repositories and begin to distribute them, then this causes problems as I described before. Distributing intermediate products is always a bad idea and systems should not be designed with this in mind or even advertise it. That has nothing to do with 'OE' or 'Debian' or any other System, its just general software developing and maintenance rules. It is ok to to this occasionally on a case by case basis, but it should not be a requirement or even an advertised feature. > ... In the rest of your email or this thread I still don't see any reason why sharing caches is necessary. Building the output from the input is fine once to fill the caches. Just to be sure that you are in fact able to build the output from the input. And I don't get why this is bad in your opinion. I get that we can trust that Debian can build their packages from their sources, so we don't need to test that, but at least test if we can still build our own packages in isar. Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de PGP key: 6FF2 E59F 00C6 BC28 31D8 64C1 1173 CB19 9808 B153 Keyserver: hkp://pool.sks-keyservers.net [-- Attachment #2: This is a digitally signed message part --] [-- Type: application/pgp-signature, Size: 833 bytes --] ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 17:34 ` Claudius Heine @ 2017-10-04 18:47 ` Alexander Smirnov 2017-10-05 8:38 ` Claudius Heine 0 siblings, 1 reply; 19+ messages in thread From: Alexander Smirnov @ 2017-10-04 18:47 UTC (permalink / raw) To: Claudius Heine, Jan Kiszka, [ext] Claudius Heine, isar-users On 10/04/2017 08:34 PM, Claudius Heine wrote: > Hi Alex, > > On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote: >> On 10/04/2017 02:57 PM, Jan Kiszka wrote: >>> On 2017-10-04 10:32, [ext] Claudius Heine wrote: >>>> Hi, >>>> >>>> On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: >>>>> Hello everybody, >>>>> >>>>> this series introduces basic binary caching for Isar. >>>>> >>>>> There is a new layer: meta-isar-bin which is intended to be the >>>>> binary >>>>> cache. All the packages that are built in Isar are stored in >>>>> this >>>>> cache using reprepro. >>>>> >>>>> Having the separate layer makes possible to manage this cache >>>>> separately >>>>> from the Isar build tree. So once you have built your packages, >>>>> you could >>>>> re-use this cache for further builds. >>>> >>>> We still have to talk about your requirement for the cache to be >>>> sharable and your resulting implementation of it as a meta layer. >>>> >>>> Since I still don't see the benefit of a sharable cache and think >>>> that >>>> putting binary packages into some kind of strange meta layer is >>>> not the >>>> right solution and could potentially create more problems that is >>>> solves >>>> because its very much in conflict with the fundamental idea of a >>>> meta >>>> layer. >> >> The last statement is a bit strange for me. The idea of meta layers >> is >> to split software stack into logical layers that could be >> enabled/disabled on demand. Having binaries or recipes doesn't >> matter >> here, it's just the way how software is provided to build. >> >>> >>> Indeed, state like this (disclaimer: i didn't look into all >>> details), it >>> sounds like a weird architecture to me as well. Thinking of an >>> artifact >>> cache, I would rather expect something like OE's sstate that one >>> may >>> preserve across builds, share between devs, or whatever. If it's >>> there, >>> the build system consults it, if not, it rebuilds. But sstate is >>> not a >>> layer, for some good reasons. >>> >> >> Let me again summarize major points here. Isar is the build system >> designed to work with binary packages. That's the key feature of >> this >> product and main difference from OE-like systems. The Isar >> architecture >> assumes to be designed around binary packages, what provides >> absolutely >> different approach how binary images could be generated using >> bitbake. >> >> According to the Claudius's vision, that everything should always be >> built from sources - Isar is not the best option here, Yocto and OE >> are >> much better, because they were initially designed for this work. >> Attempts to apply OE design and philosophy to Isar could have >> negative >> impact and limit possible ways to implement custom features, because >> OE >> wasn't designed for this features. >> Also I wonder why we are able to use binary Debian packages, while >> the >> rest of software should be built from sources. > > You are still not understanding me. I never said you have to build > every package from source code in isar. Instead I mean that you have to > build the root filesystem / images / packages ... ( every output of > isar) from all the available input of isar (upstream debian > repositories, meta-layers, source code repositories, etc.). Yes, and I want to add my custom apt repo to the Isar, even if I have source code for packages in it. > > If you create intermediate products in the form of caches, then that is > ok. But if you put those intermediate products into repositories and > begin to distribute them, then this causes problems as I described > before. > If you mean version-control repository, then nobody asks you to do this. If you mean apt repository, Isar should be able to export it with your product (application layer), like I've mentioned with examples in my mail. > Distributing intermediate products is always a bad idea and systems > should not be designed with this in mind or even advertise it. That has > nothing to do with 'OE' or 'Debian' or any other System, its just > general software developing and maintenance rules. That's contradictory sentences for me. Debian distributes both: - Complete iso, USB etc... images - Apt repository And both are intermediate in terms of your custom product. So if you want to develop package for Debian, you do not need to build whole Debian from the scratch, you likely use intermediate product instead - install base system and tools from binaries. Isar is based on intermediate product, i.e. Debian apt repository. So I still don't understand, why you consider that 'apt-get install' from Debian repository is OK, but 'apt-get install' from custom repository is a bad idea? > > It is ok to to this occasionally on a case by case basis, but it should > not be a requirement or even an advertised feature. > >> ... > > In the rest of your email or this thread I still don't see any reason > why sharing caches is necessary. > > Building the output from the input is fine once to fill the caches. > Just to be sure that you are in fact able to build the output from the > input. And I don't get why this is bad in your opinion. > Because I don't want to waste the time for the work that has been already done and tested. Just for sure, the release builds should be run from the scratch. And integration testing should be performed. Building everything from the scratch only guarantees that your package could be built, nothing more. But I'd prefer to delegate this task to CI. > I get that we can trust that Debian can build their packages from their > sources, so we don't need to test that, but at least test if we can > still build our own packages in isar. Sorry, didn't get this. Alex ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: [PATCH v4 0/4] Basic binary cache implementation 2017-10-04 18:47 ` Alexander Smirnov @ 2017-10-05 8:38 ` Claudius Heine 0 siblings, 0 replies; 19+ messages in thread From: Claudius Heine @ 2017-10-05 8:38 UTC (permalink / raw) To: Alexander Smirnov, Claudius Heine, Jan Kiszka, isar-users Hi Alex, On 10/04/2017 08:47 PM, Alexander Smirnov wrote: > On 10/04/2017 08:34 PM, Claudius Heine wrote: >> Hi Alex, >> >> On Wed, 2017-10-04 at 17:29 +0300, Alexander Smirnov wrote: >>> On 10/04/2017 02:57 PM, Jan Kiszka wrote: >>>> On 2017-10-04 10:32, [ext] Claudius Heine wrote: >>>>> Hi, >>>>> >>>>> On 10/0 '2/2017 05:45 PM, Alexander Smirnov wrote: >>>>>> Hello everybody, >>>>>> >>>>>> this series introduces basic binary caching for Isar. >>>>>> >>>>>> There is a new layer: meta-isar-bin which is intended to be the >>>>>> binary >>>>>> cache. All the packages that are built in Isar are stored in >>>>>> this >>>>>> cache using reprepro. >>>>>> >>>>>> Having the separate layer makes possible to manage this cache >>>>>> separately >>>>>> from the Isar build tree. So once you have built your packages, >>>>>> you could >>>>>> re-use this cache for further builds. >>>>> >>>>> We still have to talk about your requirement for the cache to be >>>>> sharable and your resulting implementation of it as a meta layer. >>>>> >>>>> Since I still don't see the benefit of a sharable cache and think >>>>> that >>>>> putting binary packages into some kind of strange meta layer is >>>>> not the >>>>> right solution and could potentially create more problems that is >>>>> solves >>>>> because its very much in conflict with the fundamental idea of a >>>>> meta >>>>> layer. >>> >>> The last statement is a bit strange for me. The idea of meta layers >>> is >>> to split software stack into logical layers that could be >>> enabled/disabled on demand. Having binaries or recipes doesn't >>> matter >>> here, it's just the way how software is provided to build. >>> >>>> >>>> Indeed, state like this (disclaimer: i didn't look into all >>>> details), it >>>> sounds like a weird architecture to me as well. Thinking of an >>>> artifact >>>> cache, I would rather expect something like OE's sstate that one >>>> may >>>> preserve across builds, share between devs, or whatever. If it's >>>> there, >>>> the build system consults it, if not, it rebuilds. But sstate is >>>> not a >>>> layer, for some good reasons. >>>> >>> >>> Let me again summarize major points here. Isar is the build system >>> designed to work with binary packages. That's the key feature of >>> this >>> product and main difference from OE-like systems. The Isar >>> architecture >>> assumes to be designed around binary packages, what provides >>> absolutely >>> different approach how binary images could be generated using >>> bitbake. >>> >>> According to the Claudius's vision, that everything should always be >>> built from sources - Isar is not the best option here, Yocto and OE >>> are >>> much better, because they were initially designed for this work. >>> Attempts to apply OE design and philosophy to Isar could have >>> negative >>> impact and limit possible ways to implement custom features, because >>> OE >>> wasn't designed for this features. >>> Also I wonder why we are able to use binary Debian packages, while >>> the >>> rest of software should be built from sources. >> >> You are still not understanding me. I never said you have to build >> every package from source code in isar. Instead I mean that you have to >> build the root filesystem / images / packages ... ( every output of >> isar) from all the available input of isar (upstream debian >> repositories, meta-layers, source code repositories, etc.). > > Yes, and I want to add my custom apt repo to the Isar, even if I have > source code for packages in it. > >> >> If you create intermediate products in the form of caches, then that is >> ok. But if you put those intermediate products into repositories and >> begin to distribute them, then this causes problems as I described >> before. >> > > If you mean version-control repository, then nobody asks you to do this. > If you mean apt repository, Isar should be able to export it with your > product (application layer), like I've mentioned with examples in my mail. > >> Distributing intermediate products is always a bad idea and systems >> should not be designed with this in mind or even advertise it. That has >> nothing to do with 'OE' or 'Debian' or any other System, its just >> general software developing and maintenance rules. > > That's contradictory sentences for me. Debian distributes both: > > - Complete iso, USB etc... images > - Apt repository > > And both are intermediate in terms of your custom product. But thats a different feature. Yes eventually isar should export a tarball of the repository to the deploy dir or similar. But that is not what this patchset is about. > > So if you want to develop package for Debian, you do not need to build > whole Debian from the scratch, you likely use intermediate product > instead - install base system and tools from binaries. > > Isar is based on intermediate product, i.e. Debian apt repository. The Debian repository is not a intermediate product for isar! But the resulting apt repository that is created by isar is a intermediate or final product of isar because it is generated by isar, therefor it should not be used as the input of a build process! But I am ok with it being used as a local cache! The difference is that the cache is not normally distributed to the user while the input is normally distributed to the user. Cache is internal to the build process while output and input is external. > > So I still don't understand, why you consider that 'apt-get install' > from Debian repository is OK, but 'apt-get install' from custom > repository is a bad idea? This discussion is frankly moving from frustrating to slightly infuriating. Implementing the possibility to use additional apt repositories that are added to the multistrap.conf and are defined in a bitbake variable is also a totally fine feature. But that is also not what this patchset is about. I don't appreciate you putting words in my mouth and using straw men arguments and moving the goalpost is not how we can discuss these things constructively. > >> >> It is ok to to this occasionally on a case by case basis, but it should >> not be a requirement or even an advertised feature. >> >>> ... >> >> In the rest of your email or this thread I still don't see any reason >> why sharing caches is necessary. >> >> Building the output from the input is fine once to fill the caches. >> Just to be sure that you are in fact able to build the output from the >> input. And I don't get why this is bad in your opinion. >> > > Because I don't want to waste the time for the work that has been > already done and tested. > > Just for sure, the release builds should be run from the scratch. And > integration testing should be performed. > > Building everything from the scratch only guarantees that your package > could be built, nothing more. But I'd prefer to delegate this task to CI. > >> I get that we can trust that Debian can build their packages from their >> sources, so we don't need to test that, but at least test if we can >> still build our own packages in isar. > > Sorry, didn't get this. Yes I see, and I don't know how I can explain it any simpler. Here is the main bad idea of this patch: - Using a meta-layer as a place to dump isar generated output at. meta-layers should only contain input data. All my other arguments, that you don't seem to understand, are trying to explain why this is a bad idea. Also I would really like to have a package cache, the ability to export the generated repository and the ability to used additional repositories. But the way this is integrated mixes input and output of the build system and makes many things unclear. So those 3 features should be handled separately. Claudius -- DENX Software Engineering GmbH, Managing Director: Wolfgang Denk HRB 165235 Munich, Office: Kirchenstr.5, D-82194 Groebenzell, Germany Phone: (+49)-8142-66989-54 Fax: (+49)-8142-66989-80 Email: ch@denx.de ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2017-10-05 8:38 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-10-02 15:45 [PATCH v4 0/4] Basic binary cache implementation Alexander Smirnov 2017-10-02 15:45 ` [PATCH 1/4] meta-isar-bin: Add reprepro configs Alexander Smirnov 2017-10-02 15:45 ` [PATCH 2/4] meta-isar-bin: Generate cache repos Alexander Smirnov 2017-10-04 9:03 ` Henning Schild 2017-10-04 10:57 ` Alexander Smirnov 2017-10-04 14:09 ` Henning Schild 2017-10-02 15:45 ` [PATCH 3/4] meta-isar-bin: Populate cache Alexander Smirnov 2017-10-02 15:45 ` [PATCH 4/4] meta-isar-bin: Install packages via multistrap Alexander Smirnov 2017-10-04 9:05 ` Henning Schild 2017-10-04 10:59 ` Alexander Smirnov 2017-10-04 8:32 ` [PATCH v4 0/4] Basic binary cache implementation Claudius Heine 2017-10-04 11:57 ` Jan Kiszka 2017-10-04 14:29 ` Alexander Smirnov 2017-10-04 15:40 ` Henning Schild 2017-10-04 16:50 ` Alexander Smirnov 2017-10-04 16:58 ` Henning Schild 2017-10-04 17:34 ` Claudius Heine 2017-10-04 18:47 ` Alexander Smirnov 2017-10-05 8:38 ` Claudius Heine
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox