[PATCH] fetch/git: add support for disabling shared clones on unpack

[PATCH] fetch/git: add support for disabling shared clones on unpack

[Henning Schild]

By default the unpacker will create a "shared" clone when cloning from
the DL_DIR to the WORKDIR. This patch introduces an option to control
that behaviour. Probably something that hardly anyone would want to do.

Imagine some recipe steps are executed in a namespace that is different
from the one your downloader and unpacker ran in. (chroot) Because a
"shared" clone has an absolute reference to its "alternate" you now
have to make that "alternate" visible in that new namespace (chroot) at
the exact place.

With this patch you can unpack "noshared" and get a stand-alone copy.
This copy will also work if the "alternate" is not visible or existant.

Signed-off-by: Henning Schild <henning.schild@siemens.com>
---
 lib/bb/fetch2/git.py | 13 ++++++++++++-
 1 file changed, 12 insertions(+), 1 deletion(-)

diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
index 5ef8cd69..7b7f02b2 100644
--- a/lib/bb/fetch2/git.py
+++ b/lib/bb/fetch2/git.py
@@ -53,6 +53,13 @@ Supported SRC_URI options are:
    For local git:// urls to use the current branch HEAD as the revision for use with
    AUTOREV. Implies nobranch.
 
+- noshared
+   When unpacking do not clone with the parameter "--shared". This option will
+   allow the unpacked copy to work stand-alone i.e. if your recipe runs in a
+   chroot where the "alternate" can not be found. Setting this will increase
+   the unpack-time and the disk-usage.
+   The default is "0", set noshared=1 if needed.
+
 """
 
 #Copyright (C) 2005 Richard Purdie
@@ -159,6 +166,8 @@ class Git(FetchMethod):
 
         ud.nobranch = ud.parm.get("nobranch","0") == "1"
 
+        ud.noshared = ud.parm.get("noshared","0") == "1"
+
         # usehead implies nobranch
         ud.usehead = ud.parm.get("usehead","0") == "1"
         if ud.usehead:
@@ -176,7 +185,9 @@ class Git(FetchMethod):
         if len(branches) != len(ud.names):
             raise bb.fetch2.ParameterError("The number of name and branch parameters is not balanced", ud.url)
 
-        ud.cloneflags = "-s -n"
+        ud.cloneflags = "-n"
+        if not ud.noshared:
+            ud.cloneflags += " -s"
         if ud.bareclone:
             ud.cloneflags += " --mirror"
 
-- 
2.13.6

Re: [PATCH] fetch/git: add support for disabling shared clones on unpack

[Henning Schild]

Sorry for the noise, but the mailinglist of bitbake is more like
"closedembedded". My messages where never approved and i had to
subscribe ...

Am Wed, 20 Dec 2017 17:42:09 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> By default the unpacker will create a "shared" clone when cloning from
> the DL_DIR to the WORKDIR. This patch introduces an option to control
> that behaviour. Probably something that hardly anyone would want to
> do.
> 
> Imagine some recipe steps are executed in a namespace that is
> different from the one your downloader and unpacker ran in. (chroot)
> Because a "shared" clone has an absolute reference to its "alternate"
> you now have to make that "alternate" visible in that new namespace
> (chroot) at the exact place.
> 
> With this patch you can unpack "noshared" and get a stand-alone copy.
> This copy will also work if the "alternate" is not visible or
> existant.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  lib/bb/fetch2/git.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 5ef8cd69..7b7f02b2 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -53,6 +53,13 @@ Supported SRC_URI options are:
>     For local git:// urls to use the current branch HEAD as the
> revision for use with AUTOREV. Implies nobranch.
>  
> +- noshared
> +   When unpacking do not clone with the parameter "--shared". This
> option will
> +   allow the unpacked copy to work stand-alone i.e. if your recipe
> runs in a
> +   chroot where the "alternate" can not be found. Setting this will
> increase
> +   the unpack-time and the disk-usage.
> +   The default is "0", set noshared=1 if needed.
> +
>  """
>  
>  #Copyright (C) 2005 Richard Purdie
> @@ -159,6 +166,8 @@ class Git(FetchMethod):
>  
>          ud.nobranch = ud.parm.get("nobranch","0") == "1"
>  
> +        ud.noshared = ud.parm.get("noshared","0") == "1"
> +
>          # usehead implies nobranch
>          ud.usehead = ud.parm.get("usehead","0") == "1"
>          if ud.usehead:
> @@ -176,7 +185,9 @@ class Git(FetchMethod):
>          if len(branches) != len(ud.names):
>              raise bb.fetch2.ParameterError("The number of name and
> branch parameters is not balanced", ud.url) 
> -        ud.cloneflags = "-s -n"
> +        ud.cloneflags = "-n"
> +        if not ud.noshared:
> +            ud.cloneflags += " -s"
>          if ud.bareclone:
>              ud.cloneflags += " --mirror"
>  

Re: [PATCH] fetch/git: add support for disabling shared clones on unpack

[Henning Schild]

We actually ran into that issue on a build system called Isar.

https://github.com/ilbers/isar

This build system executes some tasks in a chroot. In this chroot a
"shared" clone will not work because it expects its "alternate" to be
at the very same location it was outside the chroot. We considered
several hacks in Isar
- patching the alternates before and after chroot
- mounting the "alternate" to the exact location in chroot
None of this is really nice, so we decided to try and do something
about it upstream.

regards,
Henning

Am Wed, 20 Dec 2017 17:42:09 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> By default the unpacker will create a "shared" clone when cloning from
> the DL_DIR to the WORKDIR. This patch introduces an option to control
> that behaviour. Probably something that hardly anyone would want to
> do.
> 
> Imagine some recipe steps are executed in a namespace that is
> different from the one your downloader and unpacker ran in. (chroot)
> Because a "shared" clone has an absolute reference to its "alternate"
> you now have to make that "alternate" visible in that new namespace
> (chroot) at the exact place.
> 
> With this patch you can unpack "noshared" and get a stand-alone copy.
> This copy will also work if the "alternate" is not visible or
> existant.
> 
> Signed-off-by: Henning Schild <henning.schild@siemens.com>
> ---
>  lib/bb/fetch2/git.py | 13 ++++++++++++-
>  1 file changed, 12 insertions(+), 1 deletion(-)
> 
> diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> index 5ef8cd69..7b7f02b2 100644
> --- a/lib/bb/fetch2/git.py
> +++ b/lib/bb/fetch2/git.py
> @@ -53,6 +53,13 @@ Supported SRC_URI options are:
>     For local git:// urls to use the current branch HEAD as the
> revision for use with AUTOREV. Implies nobranch.
>  
> +- noshared
> +   When unpacking do not clone with the parameter "--shared". This
> option will
> +   allow the unpacked copy to work stand-alone i.e. if your recipe
> runs in a
> +   chroot where the "alternate" can not be found. Setting this will
> increase
> +   the unpack-time and the disk-usage.
> +   The default is "0", set noshared=1 if needed.
> +
>  """
>  
>  #Copyright (C) 2005 Richard Purdie
> @@ -159,6 +166,8 @@ class Git(FetchMethod):
>  
>          ud.nobranch = ud.parm.get("nobranch","0") == "1"
>  
> +        ud.noshared = ud.parm.get("noshared","0") == "1"
> +
>          # usehead implies nobranch
>          ud.usehead = ud.parm.get("usehead","0") == "1"
>          if ud.usehead:
> @@ -176,7 +185,9 @@ class Git(FetchMethod):
>          if len(branches) != len(ud.names):
>              raise bb.fetch2.ParameterError("The number of name and
> branch parameters is not balanced", ud.url) 
> -        ud.cloneflags = "-s -n"
> +        ud.cloneflags = "-n"
> +        if not ud.noshared:
> +            ud.cloneflags += " -s"
>          if ud.bareclone:
>              ud.cloneflags += " --mirror"
>  

Re: [PATCH] fetch/git: add support for disabling shared clones on unpack

[Henning Schild]

Bump! Could i please get a review on this patch?

regards,
Henning

Am Wed, 20 Dec 2017 17:45:38 +0100
schrieb Henning Schild <henning.schild@siemens.com>:

> We actually ran into that issue on a build system called Isar.
> 
> https://github.com/ilbers/isar
> 
> This build system executes some tasks in a chroot. In this chroot a
> "shared" clone will not work because it expects its "alternate" to be
> at the very same location it was outside the chroot. We considered
> several hacks in Isar
> - patching the alternates before and after chroot
> - mounting the "alternate" to the exact location in chroot
> None of this is really nice, so we decided to try and do something
> about it upstream.
> 
> regards,
> Henning
> 
> Am Wed, 20 Dec 2017 17:42:09 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
> > By default the unpacker will create a "shared" clone when cloning
> > from the DL_DIR to the WORKDIR. This patch introduces an option to
> > control that behaviour. Probably something that hardly anyone would
> > want to do.
> > 
> > Imagine some recipe steps are executed in a namespace that is
> > different from the one your downloader and unpacker ran in. (chroot)
> > Because a "shared" clone has an absolute reference to its
> > "alternate" you now have to make that "alternate" visible in that
> > new namespace (chroot) at the exact place.
> > 
> > With this patch you can unpack "noshared" and get a stand-alone
> > copy. This copy will also work if the "alternate" is not visible or
> > existant.
> > 
> > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > ---
> >  lib/bb/fetch2/git.py | 13 ++++++++++++-
> >  1 file changed, 12 insertions(+), 1 deletion(-)
> > 
> > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> > index 5ef8cd69..7b7f02b2 100644
> > --- a/lib/bb/fetch2/git.py
> > +++ b/lib/bb/fetch2/git.py
> > @@ -53,6 +53,13 @@ Supported SRC_URI options are:
> >     For local git:// urls to use the current branch HEAD as the
> > revision for use with AUTOREV. Implies nobranch.
> >  
> > +- noshared
> > +   When unpacking do not clone with the parameter "--shared". This
> > option will
> > +   allow the unpacked copy to work stand-alone i.e. if your recipe
> > runs in a
> > +   chroot where the "alternate" can not be found. Setting this will
> > increase
> > +   the unpack-time and the disk-usage.
> > +   The default is "0", set noshared=1 if needed.
> > +
> >  """
> >  
> >  #Copyright (C) 2005 Richard Purdie
> > @@ -159,6 +166,8 @@ class Git(FetchMethod):
> >  
> >          ud.nobranch = ud.parm.get("nobranch","0") == "1"
> >  
> > +        ud.noshared = ud.parm.get("noshared","0") == "1"
> > +
> >          # usehead implies nobranch
> >          ud.usehead = ud.parm.get("usehead","0") == "1"
> >          if ud.usehead:
> > @@ -176,7 +185,9 @@ class Git(FetchMethod):
> >          if len(branches) != len(ud.names):
> >              raise bb.fetch2.ParameterError("The number of name and
> > branch parameters is not balanced", ud.url) 
> > -        ud.cloneflags = "-s -n"
> > +        ud.cloneflags = "-n"
> > +        if not ud.noshared:
> > +            ud.cloneflags += " -s"
> >          if ud.bareclone:
> >              ud.cloneflags += " --mirror"
> >    
> 

Re: [bitbake-devel] [PATCH] fetch/git: add support for disabling shared clones on unpack

[Henning Schild]

Hey,

this is still pretty much relevant to Isar, where an unpack with git
goes into a chroot. Having a way not clone "shared" would really help
us in Isar.

Any comments? Except for this one maybe not applying anymore, did not
try.

regards,
Henning

Am Tue, 30 Jan 2018 13:18:05 +0100
schrieb "[ext] Henning Schild" <henning.schild@siemens.com>:

> Bump! Could i please get a review on this patch?
> 
> regards,
> Henning
> 
> Am Wed, 20 Dec 2017 17:45:38 +0100
> schrieb Henning Schild <henning.schild@siemens.com>:
> 
> > We actually ran into that issue on a build system called Isar.
> > 
> > https://github.com/ilbers/isar
> > 
> > This build system executes some tasks in a chroot. In this chroot a
> > "shared" clone will not work because it expects its "alternate" to
> > be at the very same location it was outside the chroot. We
> > considered several hacks in Isar
> > - patching the alternates before and after chroot
> > - mounting the "alternate" to the exact location in chroot
> > None of this is really nice, so we decided to try and do something
> > about it upstream.
> > 
> > regards,
> > Henning
> > 
> > Am Wed, 20 Dec 2017 17:42:09 +0100
> > schrieb Henning Schild <henning.schild@siemens.com>:
> >   
> > > By default the unpacker will create a "shared" clone when cloning
> > > from the DL_DIR to the WORKDIR. This patch introduces an option to
> > > control that behaviour. Probably something that hardly anyone
> > > would want to do.
> > > 
> > > Imagine some recipe steps are executed in a namespace that is
> > > different from the one your downloader and unpacker ran in.
> > > (chroot) Because a "shared" clone has an absolute reference to its
> > > "alternate" you now have to make that "alternate" visible in that
> > > new namespace (chroot) at the exact place.
> > > 
> > > With this patch you can unpack "noshared" and get a stand-alone
> > > copy. This copy will also work if the "alternate" is not visible
> > > or existant.
> > > 
> > > Signed-off-by: Henning Schild <henning.schild@siemens.com>
> > > ---
> > >  lib/bb/fetch2/git.py | 13 ++++++++++++-
> > >  1 file changed, 12 insertions(+), 1 deletion(-)
> > > 
> > > diff --git a/lib/bb/fetch2/git.py b/lib/bb/fetch2/git.py
> > > index 5ef8cd69..7b7f02b2 100644
> > > --- a/lib/bb/fetch2/git.py
> > > +++ b/lib/bb/fetch2/git.py
> > > @@ -53,6 +53,13 @@ Supported SRC_URI options are:
> > >     For local git:// urls to use the current branch HEAD as the
> > > revision for use with AUTOREV. Implies nobranch.
> > >  
> > > +- noshared
> > > +   When unpacking do not clone with the parameter "--shared".
> > > This option will
> > > +   allow the unpacked copy to work stand-alone i.e. if your
> > > recipe runs in a
> > > +   chroot where the "alternate" can not be found. Setting this
> > > will increase
> > > +   the unpack-time and the disk-usage.
> > > +   The default is "0", set noshared=1 if needed.
> > > +
> > >  """
> > >  
> > >  #Copyright (C) 2005 Richard Purdie
> > > @@ -159,6 +166,8 @@ class Git(FetchMethod):
> > >  
> > >          ud.nobranch = ud.parm.get("nobranch","0") == "1"
> > >  
> > > +        ud.noshared = ud.parm.get("noshared","0") == "1"
> > > +
> > >          # usehead implies nobranch
> > >          ud.usehead = ud.parm.get("usehead","0") == "1"
> > >          if ud.usehead:
> > > @@ -176,7 +185,9 @@ class Git(FetchMethod):
> > >          if len(branches) != len(ud.names):
> > >              raise bb.fetch2.ParameterError("The number of name
> > > and branch parameters is not balanced", ud.url) 
> > > -        ud.cloneflags = "-s -n"
> > > +        ud.cloneflags = "-n"
> > > +        if not ud.noshared:
> > > +            ud.cloneflags += " -s"
> > >          if ud.bareclone:
> > >              ud.cloneflags += " --mirror"
> > >      
> >   
> 

Re: [bitbake-devel] [PATCH] fetch/git: add support for disabling shared clones on unpack

[Richard Purdie]

Hi,

On Mon, 2021-04-12 at 17:05 +0200, Henning Schild wrote:
> this is still pretty much relevant to Isar, where an unpack with git
> goes into a chroot. Having a way not clone "shared" would really help
> us in Isar.
> 
> Any comments? Except for this one maybe not applying anymore, did not
> try.

I'm torn on this, I can see the use case but it isn't one we've typically
run into. I can see how it is Isar specific.

I'm hesitant as the more "forks" in codepath we add to the fetcher, the 
harder it becomes to test and maintain.

Would you be able to add some tests for this to the testsuite 
(bitbake-selftest)? If we have some tests we could probably add something.

I did also wonder whether this should be a per url parmeter or a global 
fetcher behaviour option? You'd end up adding this to all urls in your 
usecase?


Cheers,

Richard

Re: [bitbake-devel] [PATCH] fetch/git: add support for disabling shared clones on unpack

[Henning Schild]

Am Tue, 13 Apr 2021 11:23:00 +0100
schrieb Richard Purdie <richard.purdie@linuxfoundation.org>:

> Hi,
> 
> On Mon, 2021-04-12 at 17:05 +0200, Henning Schild wrote:
> > this is still pretty much relevant to Isar, where an unpack with git
> > goes into a chroot. Having a way not clone "shared" would really
> > help us in Isar.
> > 
> > Any comments? Except for this one maybe not applying anymore, did
> > not try.  
> 
> I'm torn on this, I can see the use case but it isn't one we've
> typically run into. I can see how it is Isar specific.
> 
> I'm hesitant as the more "forks" in codepath we add to the fetcher,
> the harder it becomes to test and maintain.

That is clear, but thanks for considering Isar even though it is not
the biggest user of bitbake.

> Would you be able to add some tests for this to the testsuite 
> (bitbake-selftest)? If we have some tests we could probably add
> something.

Sure thing, if that increases the chance of a merge.

> I did also wonder whether this should be a per url parmeter or a
> global fetcher behaviour option? You'd end up adding this to all urls
> in your usecase?

That is a good point, yes we would be forced to set that per url. I
think i will go for 
  d.getVar("BB_GIT_NOSHARE") == "1"
in v2. A variable isar can set in its bitbake.conf.

Thanks,
Henning

> 
> Cheers,
> 
> Richard
> 
>