[PATCH] Make sure git repos are usable inside buildchroot

[PATCH] Make sure git repos are usable inside buildchroot

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

bitbake clones git repos into the package workdir in shared mode. This
creates an alternates link to the original download folder. However,
that path is invalid inside buildchroot. This disturbs package builds
which try to derive information from the git repo, such as tags or the
delta to the last release.

Bind-mount the downloads folder and patch the alternates file to restore
the git repo access inside buildchroot.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Applies on top of "Move buildchroot mounts into dpkg-base class", v3.

This obsoletes the bitbake patch Henning developed to disable sharing.

Tested via jailhouse-images (required changes not yet pushed) where the
Jailhouse recipe makes use of this.

 meta/classes/dpkg-base.bbclass                   | 10 ++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb |  1 +
 2 files changed, 11 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 895ff26..241dc2c 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -1,6 +1,15 @@
 # This software is a part of ISAR.
 # Copyright (C) 2017 Siemens AG
 
+do_adjust_git() {
+    if [ -f ${WORKDIR}/${S}/.git/objects/info/alternates ]; then
+        sed -i ${WORKDIR}/${S}/.git/objects/info/alternates \
+            -e 's|${DL_DIR}|/downloads|'
+    fi
+}
+
+addtask adjust_git after do_unpack before do_build
+
 # Add dependency from buildchroot creation
 do_build[depends] = "buildchroot:do_build"
 
@@ -30,6 +39,7 @@ do_build() {
     sudo flock ${MOUNT_LOCKFILE} -c ' \
         if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
             mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \
+            mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads; \
             mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \
             mount -t proc none ${BUILDCHROOT_DIR}/proc; \
         fi'
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 62979f5..9b4e7d3 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -38,6 +38,7 @@ do_build() {
 
     sudo rm -rf ${BUILDCHROOT_DIR}
     mkdir -p ${BUILDCHROOT_DIR}/isar-apt
+    mkdir -p ${BUILDCHROOT_DIR}/downloads
     mkdir -p ${BUILDCHROOT_DIR}/dev
     mkdir -p ${BUILDCHROOT_DIR}/proc
     mkdir -p ${BUILDCHROOT_DIR}/sys
-- 
2.13.6

Re: [PATCH] Make sure git repos are usable inside buildchroot

[Henning Schild]

There is a similar patch from Alex floating around. My opinion is
still, this should be fixed upstream and we can wait. After all nobody
forces anyone to use git and most git-servers should offer tarballs,
which are probably often faster than git.

I would rather mark git as broken in the fetcher, but that might just
me a result of me not liking git at all ;).

Henning

Am Wed, 14 Feb 2018 10:14:18 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> bitbake clones git repos into the package workdir in shared mode. This
> creates an alternates link to the original download folder. However,
> that path is invalid inside buildchroot. This disturbs package builds
> which try to derive information from the git repo, such as tags or the
> delta to the last release.
> 
> Bind-mount the downloads folder and patch the alternates file to
> restore the git repo access inside buildchroot.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> Applies on top of "Move buildchroot mounts into dpkg-base class", v3.
> 
> This obsoletes the bitbake patch Henning developed to disable sharing.
> 
> Tested via jailhouse-images (required changes not yet pushed) where
> the Jailhouse recipe makes use of this.
> 
>  meta/classes/dpkg-base.bbclass                   | 10 ++++++++++
>  meta/recipes-devtools/buildchroot/buildchroot.bb |  1 +
>  2 files changed, 11 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass
> b/meta/classes/dpkg-base.bbclass index 895ff26..241dc2c 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -1,6 +1,15 @@
>  # This software is a part of ISAR.
>  # Copyright (C) 2017 Siemens AG
>  
> +do_adjust_git() {
> +    if [ -f ${WORKDIR}/${S}/.git/objects/info/alternates ]; then
> +        sed -i ${WORKDIR}/${S}/.git/objects/info/alternates \
> +            -e 's|${DL_DIR}|/downloads|'
> +    fi
> +}
> +
> +addtask adjust_git after do_unpack before do_build
> +
>  # Add dependency from buildchroot creation
>  do_build[depends] = "buildchroot:do_build"
>  
> @@ -30,6 +39,7 @@ do_build() {
>      sudo flock ${MOUNT_LOCKFILE} -c ' \
>          if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
>              mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
> ${BUILDCHROOT_DIR}/isar-apt; \
> +            mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads; \
>              mount -t devtmpfs -o mode=0755,nosuid devtmpfs
> ${BUILDCHROOT_DIR}/dev; \ mount -t proc none ${BUILDCHROOT_DIR}/proc;
> \ fi'
> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
> 62979f5..9b4e7d3 100644 ---
> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -38,6 +38,7 @@
> do_build() { 
>      sudo rm -rf ${BUILDCHROOT_DIR}
>      mkdir -p ${BUILDCHROOT_DIR}/isar-apt
> +    mkdir -p ${BUILDCHROOT_DIR}/downloads
>      mkdir -p ${BUILDCHROOT_DIR}/dev
>      mkdir -p ${BUILDCHROOT_DIR}/proc
>      mkdir -p ${BUILDCHROOT_DIR}/sys

Re: [PATCH] Make sure git repos are usable inside buildchroot

[Alexander Smirnov]

On 02/14/2018 12:30 PM, Henning Schild wrote:
> There is a similar patch from Alex floating around. My opinion is

Thanks for the reference ;-)

> still, this should be fixed upstream and we can wait. After all nobody
> forces anyone to use git and most git-servers should offer tarballs,
> which are probably often faster than git.
> 
> I would rather mark git as broken in the fetcher, but that might just
> me a result of me not liking git at all ;).

More easy workaround is also floating on the list:

https://groups.google.com/forum/#!msg/isar-users/sAcJqSx_vqs/wetXTcMLCQAJ;context-place=forum/isar-users

Just repack objects to working dir.

Alex

> 
> Henning
> 
> Am Wed, 14 Feb 2018 10:14:18 +0100
> schrieb Jan Kiszka <jan.kiszka@siemens.com>:
> 
>> From: Jan Kiszka <jan.kiszka@siemens.com>
>>
>> bitbake clones git repos into the package workdir in shared mode. This
>> creates an alternates link to the original download folder. However,
>> that path is invalid inside buildchroot. This disturbs package builds
>> which try to derive information from the git repo, such as tags or the
>> delta to the last release.
>>
>> Bind-mount the downloads folder and patch the alternates file to
>> restore the git repo access inside buildchroot.
>>
>> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
>> ---
>>
>> Applies on top of "Move buildchroot mounts into dpkg-base class", v3.
>>
>> This obsoletes the bitbake patch Henning developed to disable sharing.
>>
>> Tested via jailhouse-images (required changes not yet pushed) where
>> the Jailhouse recipe makes use of this.
>>
>>   meta/classes/dpkg-base.bbclass                   | 10 ++++++++++
>>   meta/recipes-devtools/buildchroot/buildchroot.bb |  1 +
>>   2 files changed, 11 insertions(+)
>>
>> diff --git a/meta/classes/dpkg-base.bbclass
>> b/meta/classes/dpkg-base.bbclass index 895ff26..241dc2c 100644
>> --- a/meta/classes/dpkg-base.bbclass
>> +++ b/meta/classes/dpkg-base.bbclass
>> @@ -1,6 +1,15 @@
>>   # This software is a part of ISAR.
>>   # Copyright (C) 2017 Siemens AG
>>   
>> +do_adjust_git() {
>> +    if [ -f ${WORKDIR}/${S}/.git/objects/info/alternates ]; then
>> +        sed -i ${WORKDIR}/${S}/.git/objects/info/alternates \
>> +            -e 's|${DL_DIR}|/downloads|'
>> +    fi
>> +}
>> +
>> +addtask adjust_git after do_unpack before do_build
>> +
>>   # Add dependency from buildchroot creation
>>   do_build[depends] = "buildchroot:do_build"
>>   
>> @@ -30,6 +39,7 @@ do_build() {
>>       sudo flock ${MOUNT_LOCKFILE} -c ' \
>>           if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
>>               mount --bind ${DEPLOY_DIR_APT}/${DISTRO}
>> ${BUILDCHROOT_DIR}/isar-apt; \
>> +            mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads; \
>>               mount -t devtmpfs -o mode=0755,nosuid devtmpfs
>> ${BUILDCHROOT_DIR}/dev; \ mount -t proc none ${BUILDCHROOT_DIR}/proc;
>> \ fi'
>> diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb index
>> 62979f5..9b4e7d3 100644 ---
>> a/meta/recipes-devtools/buildchroot/buildchroot.bb +++
>> b/meta/recipes-devtools/buildchroot/buildchroot.bb @@ -38,6 +38,7 @@
>> do_build() {
>>       sudo rm -rf ${BUILDCHROOT_DIR}
>>       mkdir -p ${BUILDCHROOT_DIR}/isar-apt
>> +    mkdir -p ${BUILDCHROOT_DIR}/downloads
>>       mkdir -p ${BUILDCHROOT_DIR}/dev
>>       mkdir -p ${BUILDCHROOT_DIR}/proc
>>       mkdir -p ${BUILDCHROOT_DIR}/sys
> 

-- 
With best regards,
Alexander Smirnov

ilbers GmbH
Baierbrunner Str. 28c
D-81379 Munich
+49 (89) 122 67 24-0
http://ilbers.de/
Commercial register Munich, HRB 214197
General manager: Baurzhan Ismagulov

Re: [PATCH] Make sure git repos are usable inside buildchroot

[Jan Kiszka]

On 2018-02-14 10:43, Alexander Smirnov wrote:
> On 02/14/2018 12:30 PM, Henning Schild wrote:
>> There is a similar patch from Alex floating around. My opinion is
> 
> Thanks for the reference ;-)

Sorry, forgot about this.

> 
>> still, this should be fixed upstream and we can wait. After all nobody
>> forces anyone to use git and most git-servers should offer tarballs,
>> which are probably often faster than git.
>>
>> I would rather mark git as broken in the fetcher, but that might just
>> me a result of me not liking git at all ;).
> 
> More easy workaround is also floating on the list:
> 
> https://groups.google.com/forum/#!msg/isar-users/sAcJqSx_vqs/wetXTcMLCQAJ;context-place=forum/isar-users
> 
> 
> Just repack objects to working dir.

But this one is better than everything proposed so far because

a) it's transparent to the user (in contrast to the current bitbake
   proposal)

b) it preservers the efficiency of --shared (in contrast to repacking)

We can easily revert this patch again once bitbake upstream provides a
transparent solution as well.

Jan

-- 
Siemens AG, Corporate Technology, CT RDA IOT SES-DE
Corporate Competence Center Embedded Linux

[PATCH v2] Make sure git repos are usable inside buildchroot

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

bitbake clones git repos into the package workdir in shared mode. This
creates an alternates link to the original download folder. However,
that path is invalid inside buildchroot. This disturbs package builds
which try to derive information from the git repo, such as tags or the
delta to the last release.

Bind-mount the downloads folder and patch the alternates file to restore
the git repo access inside buildchroot.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v2:
 - set stamp-extra-info properly

 meta/classes/dpkg-base.bbclass                   | 11 +++++++++++
 meta/recipes-devtools/buildchroot/buildchroot.bb |  1 +
 2 files changed, 12 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index a4bdf16..98f25b7 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -1,6 +1,16 @@
 # This software is a part of ISAR.
 # Copyright (C) 2017 Siemens AG
 
+do_adjust_git() {
+    if [ -f ${WORKDIR}/${S}/.git/objects/info/alternates ]; then
+        sed -i ${WORKDIR}/${S}/.git/objects/info/alternates \
+            -e 's|${DL_DIR}|/downloads|'
+    fi
+}
+
+addtask adjust_git after do_unpack before do_build
+do_adjust_git[stamp-extra-info] = "${DISTRO}-${DISTRO_ARCH}"
+
 # Add dependency from buildchroot creation
 do_build[depends] = "buildchroot:do_build"
 
@@ -30,6 +40,7 @@ do_build() {
     sudo flock ${MOUNT_LOCKFILE} -c ' \
         if ! grep -q ${BUILDCHROOT_DIR}/isar-apt /proc/mounts; then \
             mount --bind ${DEPLOY_DIR_APT}/${DISTRO} ${BUILDCHROOT_DIR}/isar-apt; \
+            mount --bind ${DL_DIR} ${BUILDCHROOT_DIR}/downloads; \
             mount -t devtmpfs -o mode=0755,nosuid devtmpfs ${BUILDCHROOT_DIR}/dev; \
             mount -t proc none ${BUILDCHROOT_DIR}/proc; \
         fi'
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.bb b/meta/recipes-devtools/buildchroot/buildchroot.bb
index 504d798..520daf9 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.bb
+++ b/meta/recipes-devtools/buildchroot/buildchroot.bb
@@ -40,6 +40,7 @@ do_build() {
     # re-building a clean image:
     sudo rm -rf ${BUILDCHROOT_DIR}
     mkdir -p ${BUILDCHROOT_DIR}/isar-apt
+    mkdir -p ${BUILDCHROOT_DIR}/downloads
     mkdir -p ${BUILDCHROOT_DIR}/dev
     mkdir -p ${BUILDCHROOT_DIR}/proc
     mkdir -p ${BUILDCHROOT_DIR}/sys