Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Henning Schild <henning.schild@siemens.com>]

Am Tue, 1 Dec 2020 17:44:56 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 01.12.20 16:40, Henning Schild wrote:
> > Am Tue, 1 Dec 2020 12:41:08 +0100
> > schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> >   
> >> On 01.12.20 11:41, [ext] Raphael Lisicki wrote:  
> >>> Hi,
> >>>
> >>> the function do_adjust_git does not work if the fetcher has been
> >>> set to "gitsm" which is required if the project contains git
> >>> submodules.
> >>>
> >>> See
> >>> https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26
> >>>
> >>>
> >>> This will cause the .git/objects/info/alternates remain
> >>> unmodified, which in turn breaks git from inside the buildchroot.
> >>>    
> >>
> >> From
> >> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher:
> >>
> >> "The Git Submodules fetcher is not a complete fetcher
> >> implementation. The fetcher has known issues where it does not use
> >> the normal source mirroring infrastructure properly. [...]"
> >>
> >> For that reason already, I would strongly discourage the use of
> >> this fetcher. And that's why it was never tested with Isar (to my
> >> best knowledge).  
> > 
> > I have seen people use git submodules in isar recipes. But they are
> > hairy indeed. A big issue is whether those submodules are absolute
> > or relative to their main repo. Relative is your best bet, absolute
> > will break authentication since you might find yourself wanting to
> > clone with ssh and gitsm contains absolute https links.
> > 
> > If you control that main repo, you might want to switch to relative
> > ... if you can. Doing that breaks the "forking workflow" ... now
> > every contributor needs to fork all, or switch to absolute before
> > working.
> > 
> > The worst case would be nested submodules ...
> > 
> > I found it best to switch over to repeating the submodule stuff in
> > the recipe, naming all modules with their shas and where they
> > should be cloned. Hard to maintain ... but you can write helper
> > scripts writing ".inc" files containing the bits and using a clone
> > with initialized modules as input.
> > 
> > SRC_URI = "gitsm://foo.git;destsuffix=${P}"
> > SRCREV = "0815"
> > 
> > becomes
> > 
> > SRC_URI = "git://foo.git;destsuffix=${P};name=main
> > SRCREV_main = "0815"
> > SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1
> > SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2
> > SRCREV_module1 = "1234"
> > SRCREV_module2 = "abcd"
> > 
> > Those last 4 lines can be generated from ".gitmodules" and "git
> > ls-tree <main-repo-sha> <sm-path>" and stored in an include.
> >   
> 
> That's a better pattern, indeed.
> 
> Maybe someone wants to hack up some scripts/unroll-gitsm.sh to spit
> out those lines for a given recipe.

Coming up with a truly generic script for that would be like fixing the
gitsm fetcher, pretty hard. And if anyone wants to go ahead, fixing the
sm fetcher in bitbake would be better.

Writing a script for a single recipe is "easy", so that is maybe what
people should start with. The real problem is getting getting the url
right, it can not be taken from ".gitmodules". Because there you get
the ssh vs http(s). Absolute vs. relative ... in the worst case with
PREMIRROS. And if you are really unlucky with submodule recursion.

I currently work on a layer where we have scripts generating includes
to bump recipes. Those scripts are per recipe and very layer-specific.
There it is mainly about generating the checksums that bitbake wants in
the recipes. Bitbake could learn from gentoo, but that is another story.

Henning

> Jan
>