public inbox for isar-users@googlegroups.com
 help / color / mirror / Atom feed
* [PATCH v3 0/4] Buildchroot prepare/cleanup
@ 2017-11-26 22:00 Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 1/4] dpkg-base: Fix buildchroot dependency Alexander Smirnov
                   ` (3 more replies)
  0 siblings, 4 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-11-26 22:00 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Hi all,

the goal of this series is to intrioduce 'prepare' and 'cleanup' tasks
for buildchroot. Here they are used to mount/umount git downloads to
buildchroot. In parallel several issue were fixed.

Changes since v2:
 - Added cleanup hook: if package build fails - git folder will be unmounted.
 - Restore original alternative for packages.

With best regards,
Alex

Alexander Smirnov (4):
  dpkg-base: Fix buildchroot dependency
  image: Add do_rootfs template to image class
  buildchroot: Add prepare and cleanup tasks
  dpkg-base: Update git alternates

 meta-isar/recipes-core/images/isar-image-base.bb |  3 ---
 meta/classes/dpkg-base.bbclass                   | 16 ++++++++++++++--
 meta/classes/image.bbclass                       |  9 +++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb | 21 +++++++++++++++++++++
 4 files changed, 44 insertions(+), 5 deletions(-)

-- 
2.9.5


^ permalink raw reply	[flat|nested] 11+ messages in thread
* [PATCH v3 1/4] dpkg-base: Fix buildchroot dependency
  2017-11-26 22:00 [PATCH v3 0/4] Buildchroot prepare/cleanup Alexander Smirnov
@ 2017-11-26 22:00 ` Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 2/4] image: Add do_rootfs template to image class Alexander Smirnov
                   ` (2 subsequent siblings)
  3 siblings, 0 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-11-26 22:00 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Existing code forces recipe to depends from 'do_build' task for all
the items listed in DEPENDS, what actually makes no sense. This task
only produces binary deb package and it can't satisfy any dependencies
for another task.

Original idea was to add dependency from 'buildchroot:do_build' task,
so this patch fixes this.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 3 +--
 1 file changed, 1 insertion(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 35af6d5..4941f9b 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,8 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-DEPENDS += "buildchroot"
-do_build[deptask] = "do_build"
+do_build[depends] = "buildchroot:do_build"
 
 # Each package should have its own unique build folder, so use
 # recipe name as identifier
-- 
2.9.5


^ permalink raw reply	[flat|nested] 11+ messages in thread
* [PATCH v3 2/4] image: Add do_rootfs template to image class
  2017-11-26 22:00 [PATCH v3 0/4] Buildchroot prepare/cleanup Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 1/4] dpkg-base: Fix buildchroot dependency Alexander Smirnov
@ 2017-11-26 22:00 ` Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 3/4] buildchroot: Add prepare and cleanup tasks Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 4/4] dpkg-base: Update git alternates Alexander Smirnov
  3 siblings, 0 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-11-26 22:00 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Add 'do_rootfs' task template to the image class due to it's
mandatory task. Moreover image class already has deps from
'do_rootfs' what makes it inconsistent for now.

Also this patch removes trailing empty line.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta-isar/recipes-core/images/isar-image-base.bb | 3 ---
 meta/classes/image.bbclass                       | 8 ++++++++
 2 files changed, 8 insertions(+), 3 deletions(-)

diff --git a/meta-isar/recipes-core/images/isar-image-base.bb b/meta-isar/recipes-core/images/isar-image-base.bb
index a6906c6..32d1abd 100644
--- a/meta-isar/recipes-core/images/isar-image-base.bb
+++ b/meta-isar/recipes-core/images/isar-image-base.bb
@@ -26,7 +26,6 @@ IMAGE_PREINSTALL += "apt \
 
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}"
 
-do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 do_rootfs[dirs] = "${WORKDIR}/hooks_multistrap"
 
 do_rootfs() {
@@ -70,5 +69,3 @@ do_rootfs() {
     sudo rm "${IMAGE_ROOTFS}/${DISTRO_CONFIG_SCRIPT}"
     _do_rootfs_cleanup
 }
-
-addtask rootfs before do_build after do_populate
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 2c2bafc..7813b16 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -84,6 +84,14 @@ do_populate() {
 addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
 
+do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
+
+do_rootfs() {
+    die "No root filesystem function defined, please implement in your recipe"
+}
+
+addtask rootfs before do_build after do_populate
+
 do_copy_boot_files() {
     KERNEL_IMAGE=${@get_image_name(d, 'vmlinuz')}
     if [ -n "${KERNEL_IMAGE}" ]; then
-- 
2.9.5


^ permalink raw reply	[flat|nested] 11+ messages in thread
* [PATCH v3 3/4] buildchroot: Add prepare and cleanup tasks
  2017-11-26 22:00 [PATCH v3 0/4] Buildchroot prepare/cleanup Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 1/4] dpkg-base: Fix buildchroot dependency Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 2/4] image: Add do_rootfs template to image class Alexander Smirnov
@ 2017-11-26 22:00 ` Alexander Smirnov
  2017-11-26 22:00 ` [PATCH v3 4/4] dpkg-base: Update git alternates Alexander Smirnov
  3 siblings, 0 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-11-26 22:00 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Some packages could require builchroot filesystem tunning, for
example share Isar downloads folder with chroot filesystem.

This patch adds two tasks for buildchroot:

1. 'do_prepare': This task is executed after buildchroot rootfs
generation and before 'do_build' from packages to be built.
Some notes:
 - An individual task is required for the case when you want to rebuild
   some package, this would require to run this preparation stuff for
   existing buildchroot rootfs.
 - This task should not have stamp. This means that anytime you want to
   build something, 'do_prepare' should be executed.

2. 'do_cleanup': This task is executed after all the packages are deployed.
Some notes:
 - This task also should not have stamp.
 - This task depends from the recipes listed in IMAGE_INSTALL.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass                   |  3 ++-
 meta/classes/image.bbclass                       |  1 +
 meta/recipes-devtools/buildchroot/buildchroot.bb | 21 +++++++++++++++++++++
 3 files changed, 24 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 4941f9b..5d7f8b6 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -2,7 +2,7 @@
 # Copyright (C) 2017 Siemens AG
 
 # Add dependency from buildchroot creation
-do_build[depends] = "buildchroot:do_build"
+do_build[depends] = "buildchroot:do_prepare"
 
 # Each package should have its own unique build folder, so use
 # recipe name as identifier
@@ -24,6 +24,7 @@ do_build() {
         ret=$?
         sudo umount ${BUILDROOT} 2>/dev/null || true
         sudo rmdir ${BUILDROOT} 2>/dev/null || true
+        (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
         (exit $ret) || bb_exit_handler
     }
     trap '_do_build_cleanup' EXIT
diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 7813b16..b0f645c 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -83,6 +83,7 @@ do_populate() {
 
 addtask populate before do_build after do_unpack
 do_populate[deptask] = "do_deploy_deb"
+do_populate[depends] = "buildchroot:do_cleanup"
 
 do_rootfs[stamp-extra-info] = "${MACHINE}-${DISTRO}"
 
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 6a94733..fe88e98 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -67,7 +67,28 @@ do_build() {
     # Install package builder script
     sudo install -m 755 ${WORKDIR}/build.sh ${BUILDCHROOT_DIR}
 
+    # Create share point for downloads
+    sudo install -d ${BUILDCHROOT_DIR}/git
+
     # Configure root filesystem
     sudo chroot ${BUILDCHROOT_DIR} /configscript.sh
     _do_build_cleanup
 }
+
+do_prepare[nostamp] = "1"
+
+do_prepare() {
+    sudo mount --bind ${GITDIR} ${BUILDCHROOT_DIR}/git
+}
+
+addtask prepare after do_build
+
+DEPENDS += "${IMAGE_INSTALL}"
+do_cleanup[deptask] = "do_deploy_deb"
+do_cleanup[nostamp] = "1"
+
+do_cleanup() {
+    sudo umount ${BUILDCHROOT_DIR}/git
+}
+
+addtask cleanup after do_prepare
-- 
2.9.5


^ permalink raw reply	[flat|nested] 11+ messages in thread
* [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-11-26 22:00 [PATCH v3 0/4] Buildchroot prepare/cleanup Alexander Smirnov
                   ` (2 preceding siblings ...)
  2017-11-26 22:00 ` [PATCH v3 3/4] buildchroot: Add prepare and cleanup tasks Alexander Smirnov
@ 2017-11-26 22:00 ` Alexander Smirnov
  2017-12-11 17:56   ` Henning Schild
  2017-12-11 18:03   ` Jan Kiszka
  3 siblings, 2 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-11-26 22:00 UTC (permalink / raw)
  To: isar-users; +Cc: Alexander Smirnov

Update git alternates to work in buildchroot.

Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 12 ++++++++++++
 1 file changed, 12 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 5d7f8b6..6a78a26 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -18,12 +18,24 @@ dpkg_runbuild() {
 
 # Wrap the function dpkg_runbuild with the bind mount for buildroot
 do_build() {
+    if [ -d ${WORKDIR}/git/.git ]; then
+        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
+        REPO_PATH=$(dirname $OBJ_PATH)
+        REPO_NAME=$(basename $REPO_PATH)
+        echo "/git/$REPO_NAME/objects" > ${WORKDIR}/git/.git/objects/info/alternates
+    fi
+
     mkdir -p ${BUILDROOT}
     sudo mount --bind ${WORKDIR} ${BUILDROOT}
     _do_build_cleanup() {
         ret=$?
         sudo umount ${BUILDROOT} 2>/dev/null || true
         sudo rmdir ${BUILDROOT} 2>/dev/null || true
+
+        if [ -d ${WORKDIR}/git/.git ]; then
+            echo $OBJ_PATH > ${WORKDIR}/git/.git/objects/info/alternates
+        fi
+
         (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
         (exit $ret) || bb_exit_handler
     }
-- 
2.9.5


^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-11-26 22:00 ` [PATCH v3 4/4] dpkg-base: Update git alternates Alexander Smirnov
@ 2017-12-11 17:56   ` Henning Schild
  2017-12-11 19:45     ` Alexander Smirnov
  2017-12-11 18:03   ` Jan Kiszka
  1 sibling, 1 reply; 11+ messages in thread
From: Henning Schild @ 2017-12-11 17:56 UTC (permalink / raw)
  To: Alexander Smirnov; +Cc: isar-users

Well i thought about this again. This is a very nasty workaround for a
problem this actually is in bitbake.

https://github.com/openembedded/bitbake/blob/ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8/lib/bb/fetch2/git.py#L179

That problem should be fixed in bitbake, if they accept patches. The
"-s" needs to depend on some condition.

For Isar i would tell people to fetch tarballs from git servers and
avoid git://.

Henning

Am Mon, 27 Nov 2017 01:00:25 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> Update git alternates to work in buildchroot.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 5d7f8b6..6a78a26 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -18,12 +18,24 @@ dpkg_runbuild() {
>  
>  # Wrap the function dpkg_runbuild with the bind mount for buildroot
>  do_build() {
> +    if [ -d ${WORKDIR}/git/.git ]; then
> +        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
> +        REPO_PATH=$(dirname $OBJ_PATH)
> +        REPO_NAME=$(basename $REPO_PATH)
> +        echo "/git/$REPO_NAME/objects" >
> ${WORKDIR}/git/.git/objects/info/alternates
> +    fi
> +
>      mkdir -p ${BUILDROOT}
>      sudo mount --bind ${WORKDIR} ${BUILDROOT}
>      _do_build_cleanup() {
>          ret=$?
>          sudo umount ${BUILDROOT} 2>/dev/null || true
>          sudo rmdir ${BUILDROOT} 2>/dev/null || true
> +
> +        if [ -d ${WORKDIR}/git/.git ]; then
> +            echo $OBJ_PATH >
> ${WORKDIR}/git/.git/objects/info/alternates
> +        fi
> +
>          (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
>          (exit $ret) || bb_exit_handler
>      }


^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-11-26 22:00 ` [PATCH v3 4/4] dpkg-base: Update git alternates Alexander Smirnov
  2017-12-11 17:56   ` Henning Schild
@ 2017-12-11 18:03   ` Jan Kiszka
  2017-12-11 19:49     ` Alexander Smirnov
  1 sibling, 1 reply; 11+ messages in thread
From: Jan Kiszka @ 2017-12-11 18:03 UTC (permalink / raw)
  To: Alexander Smirnov, isar-users

On 2017-11-26 23:00, Alexander Smirnov wrote:
> Update git alternates to work in buildchroot.
> 
> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 12 ++++++++++++
>  1 file changed, 12 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 5d7f8b6..6a78a26 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -18,12 +18,24 @@ dpkg_runbuild() {
>  
>  # Wrap the function dpkg_runbuild with the bind mount for buildroot
>  do_build() {
> +    if [ -d ${WORKDIR}/git/.git ]; then
> +        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
> +        REPO_PATH=$(dirname $OBJ_PATH)
> +        REPO_NAME=$(basename $REPO_PATH)
> +        echo "/git/$REPO_NAME/objects" > ${WORKDIR}/git/.git/objects/info/alternates
> +    fi
> +
>      mkdir -p ${BUILDROOT}
>      sudo mount --bind ${WORKDIR} ${BUILDROOT}
>      _do_build_cleanup() {
>          ret=$?
>          sudo umount ${BUILDROOT} 2>/dev/null || true
>          sudo rmdir ${BUILDROOT} 2>/dev/null || true
> +
> +        if [ -d ${WORKDIR}/git/.git ]; then
> +            echo $OBJ_PATH > ${WORKDIR}/git/.git/objects/info/alternates
> +        fi
> +
>          (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
>          (exit $ret) || bb_exit_handler
>      }
> 

You can probably also amend (>>) the existing alternates with the path
that is active in the chroot so that you can also look at the repo
outside of it.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA ITP SES-DE
Corporate Competence Center Embedded Linux

^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-12-11 17:56   ` Henning Schild
@ 2017-12-11 19:45     ` Alexander Smirnov
  2017-12-12 11:49       ` Henning Schild
  0 siblings, 1 reply; 11+ messages in thread
From: Alexander Smirnov @ 2017-12-11 19:45 UTC (permalink / raw)
  To: Henning Schild; +Cc: isar-users

On 12/11/2017 08:56 PM, Henning Schild wrote:
> Well i thought about this again. This is a very nasty workaround for a
> problem this actually is in bitbake.

That's an attempt to solve here and now git-3.0 issue reported by 
Christian. So if in general it's not critical problem for now, I could 
drop this series.

> 
> https://github.com/openembedded/bitbake/blob/ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8/lib/bb/fetch2/git.py#L179
> 
> That problem should be fixed in bitbake, if they accept patches. The
> "-s" needs to depend on some condition.
> 

I'm not sure that this is the best solution. The main idea of using '-s' 
was to avoid unnecessary copying of git repos within one build, it costs 
disk space and extra time for copying (for example linux.git is about 
1GB). So usage of '-s' is quite reasonable here.

Another solution could be deprecate 'downloads' directory and fetch gits 
directly to package WORKDIR.

> For Isar i would tell people to fetch tarballs from git servers and
> avoid git://.

This should not be the requirement, because it limits Isar application 
area. Not all git repo could be fetched as tarball.

Alex

> 
> Henning
> 
> Am Mon, 27 Nov 2017 01:00:25 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> Update git alternates to work in buildchroot.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 5d7f8b6..6a78a26 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -18,12 +18,24 @@ dpkg_runbuild() {
>>   
>>   # Wrap the function dpkg_runbuild with the bind mount for buildroot
>>   do_build() {
>> +    if [ -d ${WORKDIR}/git/.git ]; then
>> +        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
>> +        REPO_PATH=$(dirname $OBJ_PATH)
>> +        REPO_NAME=$(basename $REPO_PATH)
>> +        echo "/git/$REPO_NAME/objects" >
>> ${WORKDIR}/git/.git/objects/info/alternates
>> +    fi
>> +
>>       mkdir -p ${BUILDROOT}
>>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
>>       _do_build_cleanup() {
>>           ret=$?
>>           sudo umount ${BUILDROOT} 2>/dev/null || true
>>           sudo rmdir ${BUILDROOT} 2>/dev/null || true
>> +
>> +        if [ -d ${WORKDIR}/git/.git ]; then
>> +            echo $OBJ_PATH >
>> ${WORKDIR}/git/.git/objects/info/alternates
>> +        fi
>> +
>>           (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
>>           (exit $ret) || bb_exit_handler
>>       }
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-12-11 18:03   ` Jan Kiszka
@ 2017-12-11 19:49     ` Alexander Smirnov
  0 siblings, 0 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-12-11 19:49 UTC (permalink / raw)
  To: Jan Kiszka, isar-users

Hi Jan,

On 12/11/2017 09:03 PM, Jan Kiszka wrote:
> On 2017-11-26 23:00, Alexander Smirnov wrote:
>> Update git alternates to work in buildchroot.
>>
>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>> ---
>>   meta/classes/dpkg-base.bbclass | 12 ++++++++++++
>>   1 file changed, 12 insertions(+)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
>> index 5d7f8b6..6a78a26 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -18,12 +18,24 @@ dpkg_runbuild() {
>>   
>>   # Wrap the function dpkg_runbuild with the bind mount for buildroot
>>   do_build() {
>> +    if [ -d ${WORKDIR}/git/.git ]; then
>> +        OBJ_PATH=$(cat ${WORKDIR}/git/.git/objects/info/alternates)
>> +        REPO_PATH=$(dirname $OBJ_PATH)
>> +        REPO_NAME=$(basename $REPO_PATH)
>> +        echo "/git/$REPO_NAME/objects" > ${WORKDIR}/git/.git/objects/info/alternates
>> +    fi
>> +
>>       mkdir -p ${BUILDROOT}
>>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
>>       _do_build_cleanup() {
>>           ret=$?
>>           sudo umount ${BUILDROOT} 2>/dev/null || true
>>           sudo rmdir ${BUILDROOT} 2>/dev/null || true
>> +
>> +        if [ -d ${WORKDIR}/git/.git ]; then
>> +            echo $OBJ_PATH > ${WORKDIR}/git/.git/objects/info/alternates
>> +        fi
>> +
>>           (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
>>           (exit $ret) || bb_exit_handler
>>       }
>>
> 
> You can probably also amend (>>) the existing alternates with the path
> that is active in the chroot so that you can also look at the repo
> outside of it.
> 

I already tried it before. It works, but git also reports message like:

   error: alternates XXX is broken.

Not critical, but message looks nasty and annoying, also it's stored to 
build logs. :-(

Alex

^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-12-11 19:45     ` Alexander Smirnov
@ 2017-12-12 11:49       ` Henning Schild
  2017-12-12 12:30         ` Alexander Smirnov
  0 siblings, 1 reply; 11+ messages in thread
From: Henning Schild @ 2017-12-12 11:49 UTC (permalink / raw)
  To: Alexander Smirnov; +Cc: isar-users

Am Mon, 11 Dec 2017 22:45:15 +0300
schrieb Alexander Smirnov <asmirnov@ilbers.de>:

> On 12/11/2017 08:56 PM, Henning Schild wrote:
> > Well i thought about this again. This is a very nasty workaround
> > for a problem this actually is in bitbake.  
> 
> That's an attempt to solve here and now git-3.0 issue reported by 
> Christian. So if in general it's not critical problem for now, I
> could drop this series.
> 
> > 
> > https://github.com/openembedded/bitbake/blob/ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8/lib/bb/fetch2/git.py#L179
> > 
> > That problem should be fixed in bitbake, if they accept patches. The
> > "-s" needs to depend on some condition.
> >   
> 
> I'm not sure that this is the best solution. The main idea of using
> '-s' was to avoid unnecessary copying of git repos within one build,
> it costs disk space and extra time for copying (for example linux.git
> is about 1GB). So usage of '-s' is quite reasonable here.

I know why it is there, all i am saying it should maybe not be there
unconditionally. We could probably try and fix that upstrem and
introduce BB_GIT_SHARED or a new param "shared".

And now you could say
BB_GIT_SHARED=0
SRC_URI=git:// ....git
or
SRC_URI=git:// ....git;shared=0

Let me know what you think about that idea and i will prepare patches.
If they get merged we could cherry-pick them into Isar and forget about
the alternate-patching.

We could also agree that "shared" is an important feature we want in
Isar, in that case the alternates should get patched. And we need an
updated version of the proposed patch.
Thinking about that ... again ;) This is would i would now vote for.

> Another solution could be deprecate 'downloads' directory and fetch
> gits directly to package WORKDIR.

I think having the downloads dir might eventually help with the
reproducible builds.

> 
> > For Isar i would tell people to fetch tarballs from git servers and
> > avoid git://.  
> 
> This should not be the requirement, because it limits Isar
> application area. Not all git repo could be fetched as tarball.

Agreed, i am just talking about temporary workarounds until we have a
solution. Fetching with git is something that, to me, does not sound
like a high prio feature.

Henning

> Alex
> 
> > 
> > Henning
> > 
> > Am Mon, 27 Nov 2017 01:00:25 +0300
> > schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> >   
> >> Update git alternates to work in buildchroot.
> >>
> >> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
> >> ---
> >>   meta/classes/dpkg-base.bbclass | 12 ++++++++++++
> >>   1 file changed, 12 insertions(+)
> >>
> >> diff --git a/meta/classes/dpkg-base.bbclass
> >> b/meta/classes/dpkg-base.bbclass index 5d7f8b6..6a78a26 100644
> >> --- a/meta/classes/dpkg-base.bbclass
> >> +++ b/meta/classes/dpkg-base.bbclass
> >> @@ -18,12 +18,24 @@ dpkg_runbuild() {
> >>   
> >>   # Wrap the function dpkg_runbuild with the bind mount for
> >> buildroot do_build() {
> >> +    if [ -d ${WORKDIR}/git/.git ]; then
> >> +        OBJ_PATH=$(cat
> >> ${WORKDIR}/git/.git/objects/info/alternates)
> >> +        REPO_PATH=$(dirname $OBJ_PATH)
> >> +        REPO_NAME=$(basename $REPO_PATH)
> >> +        echo "/git/$REPO_NAME/objects" >
> >> ${WORKDIR}/git/.git/objects/info/alternates
> >> +    fi
> >> +
> >>       mkdir -p ${BUILDROOT}
> >>       sudo mount --bind ${WORKDIR} ${BUILDROOT}
> >>       _do_build_cleanup() {
> >>           ret=$?
> >>           sudo umount ${BUILDROOT} 2>/dev/null || true
> >>           sudo rmdir ${BUILDROOT} 2>/dev/null || true
> >> +
> >> +        if [ -d ${WORKDIR}/git/.git ]; then
> >> +            echo $OBJ_PATH >
> >> ${WORKDIR}/git/.git/objects/info/alternates
> >> +        fi
> >> +
> >>           (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
> >>           (exit $ret) || bb_exit_handler
> >>       }  
> >   
> 


^ permalink raw reply	[flat|nested] 11+ messages in thread
* Re: [PATCH v3 4/4] dpkg-base: Update git alternates
  2017-12-12 11:49       ` Henning Schild
@ 2017-12-12 12:30         ` Alexander Smirnov
  0 siblings, 0 replies; 11+ messages in thread
From: Alexander Smirnov @ 2017-12-12 12:30 UTC (permalink / raw)
  To: Henning Schild; +Cc: isar-users



On 12/12/2017 02:49 PM, Henning Schild wrote:
> Am Mon, 11 Dec 2017 22:45:15 +0300
> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
> 
>> On 12/11/2017 08:56 PM, Henning Schild wrote:
>>> Well i thought about this again. This is a very nasty workaround
>>> for a problem this actually is in bitbake.
>>
>> That's an attempt to solve here and now git-3.0 issue reported by
>> Christian. So if in general it's not critical problem for now, I
>> could drop this series.
>>
>>>
>>> https://github.com/openembedded/bitbake/blob/ebce92bf8d71f8a6e8af1c6cf6ba335faf9d67c8/lib/bb/fetch2/git.py#L179
>>>
>>> That problem should be fixed in bitbake, if they accept patches. The
>>> "-s" needs to depend on some condition.
>>>    
>>
>> I'm not sure that this is the best solution. The main idea of using
>> '-s' was to avoid unnecessary copying of git repos within one build,
>> it costs disk space and extra time for copying (for example linux.git
>> is about 1GB). So usage of '-s' is quite reasonable here.
> 
> I know why it is there, all i am saying it should maybe not be there
> unconditionally. We could probably try and fix that upstrem and
> introduce BB_GIT_SHARED or a new param "shared".
> 
> And now you could say
> BB_GIT_SHARED=0
> SRC_URI=git:// ....git
> or
> SRC_URI=git:// ....git;shared=0

I like this one, I've already investigated supported parameters for git 
protocol, there are lot of them. So extra new one should not be a 
problem :-)

> 
> Let me know what you think about that idea and i will prepare patches.

So, if it covers your use case and you could prepare the patch, this 
would be great :-)

Alex

> If they get merged we could cherry-pick them into Isar and forget about
> the alternate-patching.
> 
> We could also agree that "shared" is an important feature we want in
> Isar, in that case the alternates should get patched. And we need an
> updated version of the proposed patch.
> Thinking about that ... again ;) This is would i would now vote for.
> 
>> Another solution could be deprecate 'downloads' directory and fetch
>> gits directly to package WORKDIR.
> 
> I think having the downloads dir might eventually help with the
> reproducible builds.
> 
>>
>>> For Isar i would tell people to fetch tarballs from git servers and
>>> avoid git://.
>>
>> This should not be the requirement, because it limits Isar
>> application area. Not all git repo could be fetched as tarball.
> 
> Agreed, i am just talking about temporary workarounds until we have a
> solution. Fetching with git is something that, to me, does not sound
> like a high prio feature.
> 
> Henning
> 
>> Alex
>>
>>>
>>> Henning
>>>
>>> Am Mon, 27 Nov 2017 01:00:25 +0300
>>> schrieb Alexander Smirnov <asmirnov@ilbers.de>:
>>>    
>>>> Update git alternates to work in buildchroot.
>>>>
>>>> Signed-off-by: Alexander Smirnov <asmirnov@ilbers.de>
>>>> ---
>>>>    meta/classes/dpkg-base.bbclass | 12 ++++++++++++
>>>>    1 file changed, 12 insertions(+)
>>>>
>>>> diff --git a/meta/classes/dpkg-base.bbclass
>>>> b/meta/classes/dpkg-base.bbclass index 5d7f8b6..6a78a26 100644
>>>> --- a/meta/classes/dpkg-base.bbclass
>>>> +++ b/meta/classes/dpkg-base.bbclass
>>>> @@ -18,12 +18,24 @@ dpkg_runbuild() {
>>>>    
>>>>    # Wrap the function dpkg_runbuild with the bind mount for
>>>> buildroot do_build() {
>>>> +    if [ -d ${WORKDIR}/git/.git ]; then
>>>> +        OBJ_PATH=$(cat
>>>> ${WORKDIR}/git/.git/objects/info/alternates)
>>>> +        REPO_PATH=$(dirname $OBJ_PATH)
>>>> +        REPO_NAME=$(basename $REPO_PATH)
>>>> +        echo "/git/$REPO_NAME/objects" >
>>>> ${WORKDIR}/git/.git/objects/info/alternates
>>>> +    fi
>>>> +
>>>>        mkdir -p ${BUILDROOT}
>>>>        sudo mount --bind ${WORKDIR} ${BUILDROOT}
>>>>        _do_build_cleanup() {
>>>>            ret=$?
>>>>            sudo umount ${BUILDROOT} 2>/dev/null || true
>>>>            sudo rmdir ${BUILDROOT} 2>/dev/null || true
>>>> +
>>>> +        if [ -d ${WORKDIR}/git/.git ]; then
>>>> +            echo $OBJ_PATH >
>>>> ${WORKDIR}/git/.git/objects/info/alternates
>>>> +        fi
>>>> +
>>>>            (exit $ret) || sudo umount ${BUILDCHROOT_DIR}/git
>>>>            (exit $ret) || bb_exit_handler
>>>>        }
>>>    
>>
>

^ permalink raw reply	[flat|nested] 11+ messages in thread
end of thread, other threads:[~2017-12-12 12:30 UTC | newest]

Thread overview: 11+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2017-11-26 22:00 [PATCH v3 0/4] Buildchroot prepare/cleanup Alexander Smirnov
2017-11-26 22:00 ` [PATCH v3 1/4] dpkg-base: Fix buildchroot dependency Alexander Smirnov
2017-11-26 22:00 ` [PATCH v3 2/4] image: Add do_rootfs template to image class Alexander Smirnov
2017-11-26 22:00 ` [PATCH v3 3/4] buildchroot: Add prepare and cleanup tasks Alexander Smirnov
2017-11-26 22:00 ` [PATCH v3 4/4] dpkg-base: Update git alternates Alexander Smirnov
2017-12-11 17:56   ` Henning Schild
2017-12-11 19:45     ` Alexander Smirnov
2017-12-12 11:49       ` Henning Schild
2017-12-12 12:30         ` Alexander Smirnov
2017-12-11 18:03   ` Jan Kiszka
2017-12-11 19:49     ` Alexander Smirnov

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox