do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Raphael Lisicki]

Hi,

the function do_adjust_git does not work if the fetcher has been set to 
"gitsm" which is required if the project contains git submodules.

See 
https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26

This will cause the .git/objects/info/alternates remain unmodified, 
which in turn breaks git from inside the buildchroot.

Raphael

Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Jan Kiszka]

On 01.12.20 11:41, [ext] Raphael Lisicki wrote:
> Hi,
> 
> the function do_adjust_git does not work if the fetcher has been set to
> "gitsm" which is required if the project contains git submodules.
> 
> See
> https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26
> 
> 
> This will cause the .git/objects/info/alternates remain unmodified,
> which in turn breaks git from inside the buildchroot.

From
https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher:

"The Git Submodules fetcher is not a complete fetcher implementation.
The fetcher has known issues where it does not use the normal source
mirroring infrastructure properly. [...]"

For that reason already, I would strongly discourage the use of this
fetcher. And that's why it was never tested with Isar (to my best
knowledge).

In general, git submodules are hairy. And specifically in the context of
distro (like Debian), their presence indicate that you rather want them
as "Build-Depends" and properly packaged.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Henning Schild]

Am Tue, 1 Dec 2020 12:41:08 +0100
schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:

> On 01.12.20 11:41, [ext] Raphael Lisicki wrote:
> > Hi,
> > 
> > the function do_adjust_git does not work if the fetcher has been
> > set to "gitsm" which is required if the project contains git
> > submodules.
> > 
> > See
> > https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26
> > 
> > 
> > This will cause the .git/objects/info/alternates remain unmodified,
> > which in turn breaks git from inside the buildchroot.  
> 
> From
> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher:
> 
> "The Git Submodules fetcher is not a complete fetcher implementation.
> The fetcher has known issues where it does not use the normal source
> mirroring infrastructure properly. [...]"
> 
> For that reason already, I would strongly discourage the use of this
> fetcher. And that's why it was never tested with Isar (to my best
> knowledge).

I have seen people use git submodules in isar recipes. But they are
hairy indeed. A big issue is whether those submodules are absolute or
relative to their main repo. Relative is your best bet, absolute will
break authentication since you might find yourself wanting to clone with
ssh and gitsm contains absolute https links.

If you control that main repo, you might want to switch to relative ...
if you can. Doing that breaks the "forking workflow" ... now every
contributor needs to fork all, or switch to absolute before working.

The worst case would be nested submodules ...

I found it best to switch over to repeating the submodule stuff in the
recipe, naming all modules with their shas and where they should be
cloned. Hard to maintain ... but you can write helper scripts writing
".inc" files containing the bits and using a clone with initialized
modules as input.

SRC_URI = "gitsm://foo.git;destsuffix=${P}"
SRCREV = "0815"

becomes

SRC_URI = "git://foo.git;destsuffix=${P};name=main
SRCREV_main = "0815"
SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1
SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2
SRCREV_module1 = "1234"
SRCREV_module2 = "abcd"

Those last 4 lines can be generated from ".gitmodules" and "git ls-tree
<main-repo-sha> <sm-path>" and stored in an include.

Henning

> In general, git submodules are hairy. And specifically in the context
> of distro (like Debian), their presence indicate that you rather want
> them as "Build-Depends" and properly packaged.
> 
> Jan
> 

Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Jan Kiszka]

On 01.12.20 16:40, Henning Schild wrote:
> Am Tue, 1 Dec 2020 12:41:08 +0100
> schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> 
>> On 01.12.20 11:41, [ext] Raphael Lisicki wrote:
>>> Hi,
>>>
>>> the function do_adjust_git does not work if the fetcher has been
>>> set to "gitsm" which is required if the project contains git
>>> submodules.
>>>
>>> See
>>> https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26
>>>
>>>
>>> This will cause the .git/objects/info/alternates remain unmodified,
>>> which in turn breaks git from inside the buildchroot.  
>>
>> From
>> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher:
>>
>> "The Git Submodules fetcher is not a complete fetcher implementation.
>> The fetcher has known issues where it does not use the normal source
>> mirroring infrastructure properly. [...]"
>>
>> For that reason already, I would strongly discourage the use of this
>> fetcher. And that's why it was never tested with Isar (to my best
>> knowledge).
> 
> I have seen people use git submodules in isar recipes. But they are
> hairy indeed. A big issue is whether those submodules are absolute or
> relative to their main repo. Relative is your best bet, absolute will
> break authentication since you might find yourself wanting to clone with
> ssh and gitsm contains absolute https links.
> 
> If you control that main repo, you might want to switch to relative ...
> if you can. Doing that breaks the "forking workflow" ... now every
> contributor needs to fork all, or switch to absolute before working.
> 
> The worst case would be nested submodules ...
> 
> I found it best to switch over to repeating the submodule stuff in the
> recipe, naming all modules with their shas and where they should be
> cloned. Hard to maintain ... but you can write helper scripts writing
> ".inc" files containing the bits and using a clone with initialized
> modules as input.
> 
> SRC_URI = "gitsm://foo.git;destsuffix=${P}"
> SRCREV = "0815"
> 
> becomes
> 
> SRC_URI = "git://foo.git;destsuffix=${P};name=main
> SRCREV_main = "0815"
> SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1
> SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2
> SRCREV_module1 = "1234"
> SRCREV_module2 = "abcd"
> 
> Those last 4 lines can be generated from ".gitmodules" and "git ls-tree
> <main-repo-sha> <sm-path>" and stored in an include.
> 

That's a better pattern, indeed.

Maybe someone wants to hack up some scripts/unroll-gitsm.sh to spit out
those lines for a given recipe.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Henning Schild]

Am Tue, 1 Dec 2020 17:44:56 +0100
schrieb Jan Kiszka <jan.kiszka@siemens.com>:

> On 01.12.20 16:40, Henning Schild wrote:
> > Am Tue, 1 Dec 2020 12:41:08 +0100
> > schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>:
> >   
> >> On 01.12.20 11:41, [ext] Raphael Lisicki wrote:  
> >>> Hi,
> >>>
> >>> the function do_adjust_git does not work if the fetcher has been
> >>> set to "gitsm" which is required if the project contains git
> >>> submodules.
> >>>
> >>> See
> >>> https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26
> >>>
> >>>
> >>> This will cause the .git/objects/info/alternates remain
> >>> unmodified, which in turn breaks git from inside the buildchroot.
> >>>    
> >>
> >> From
> >> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher:
> >>
> >> "The Git Submodules fetcher is not a complete fetcher
> >> implementation. The fetcher has known issues where it does not use
> >> the normal source mirroring infrastructure properly. [...]"
> >>
> >> For that reason already, I would strongly discourage the use of
> >> this fetcher. And that's why it was never tested with Isar (to my
> >> best knowledge).  
> > 
> > I have seen people use git submodules in isar recipes. But they are
> > hairy indeed. A big issue is whether those submodules are absolute
> > or relative to their main repo. Relative is your best bet, absolute
> > will break authentication since you might find yourself wanting to
> > clone with ssh and gitsm contains absolute https links.
> > 
> > If you control that main repo, you might want to switch to relative
> > ... if you can. Doing that breaks the "forking workflow" ... now
> > every contributor needs to fork all, or switch to absolute before
> > working.
> > 
> > The worst case would be nested submodules ...
> > 
> > I found it best to switch over to repeating the submodule stuff in
> > the recipe, naming all modules with their shas and where they
> > should be cloned. Hard to maintain ... but you can write helper
> > scripts writing ".inc" files containing the bits and using a clone
> > with initialized modules as input.
> > 
> > SRC_URI = "gitsm://foo.git;destsuffix=${P}"
> > SRCREV = "0815"
> > 
> > becomes
> > 
> > SRC_URI = "git://foo.git;destsuffix=${P};name=main
> > SRCREV_main = "0815"
> > SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1
> > SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2
> > SRCREV_module1 = "1234"
> > SRCREV_module2 = "abcd"
> > 
> > Those last 4 lines can be generated from ".gitmodules" and "git
> > ls-tree <main-repo-sha> <sm-path>" and stored in an include.
> >   
> 
> That's a better pattern, indeed.
> 
> Maybe someone wants to hack up some scripts/unroll-gitsm.sh to spit
> out those lines for a given recipe.

Coming up with a truly generic script for that would be like fixing the
gitsm fetcher, pretty hard. And if anyone wants to go ahead, fixing the
sm fetcher in bitbake would be better.

Writing a script for a single recipe is "easy", so that is maybe what
people should start with. The real problem is getting getting the url
right, it can not be taken from ".gitmodules". Because there you get
the ssh vs http(s). Absolute vs. relative ... in the worst case with
PREMIRROS. And if you are really unlucky with submodule recursion.

I currently work on a layer where we have scripts generating includes
to bump recipes. Those scripts are per recipe and very layer-specific.
There it is mainly about generating the checksums that bitbake wants in
the recipes. Bitbake could learn from gentoo, but that is another story.

Henning

> Jan
> 

Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Raphael Lisicki]

Thanks for your answers. I know well about the pains of the submodules.

> 
> SRC_URI = "gitsm://foo.git;destsuffix=${P}"
> SRCREV = "0815"
> 
> becomes
> 
> SRC_URI = "git://foo.git;destsuffix=${P};name=main
> SRCREV_main = "0815"
> SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1
> SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2
> SRCREV_module1 = "1234"
> SRCREV_module2 = "abcd"
> 

This sounds like a feasible workaround. Happily, I was able to just 
switch to the git fetcher.

Simply adding "and ud.type != 'gitsm'" would also fix this for certain 
cases (mine :D ) . Given that it is known that gitsm is not working 
properly, such a best-effort fix might also be an option.

raphael

Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher

[Jan Kiszka]

On 02.12.20 18:10, Raphael Lisicki wrote:
> Thanks for your answers. I know well about the pains of the submodules.
> 
>>
>> SRC_URI = "gitsm://foo.git;destsuffix=${P}"
>> SRCREV = "0815"
>>
>> becomes
>>
>> SRC_URI = "git://foo.git;destsuffix=${P};name=main
>> SRCREV_main = "0815"
>> SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1
>> SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2
>> SRCREV_module1 = "1234"
>> SRCREV_module2 = "abcd"
>>
> 
> This sounds like a feasible workaround. Happily, I was able to just
> switch to the git fetcher.
> 
> Simply adding "and ud.type != 'gitsm'" would also fix this for certain
> cases (mine :D ) . Given that it is known that gitsm is not working
> properly, such a best-effort fix might also be an option.
> 

Feel free to send such a trivial patch. If it makes things a bit better,
no need to not merge that.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux