* do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher @ 2020-12-01 10:41 Raphael Lisicki 2020-12-01 11:41 ` Jan Kiszka 0 siblings, 1 reply; 7+ messages in thread From: Raphael Lisicki @ 2020-12-01 10:41 UTC (permalink / raw) To: isar-users Hi, the function do_adjust_git does not work if the fetcher has been set to "gitsm" which is required if the project contains git submodules. See https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26 This will cause the .git/objects/info/alternates remain unmodified, which in turn breaks git from inside the buildchroot. Raphael ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher 2020-12-01 10:41 do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher Raphael Lisicki @ 2020-12-01 11:41 ` Jan Kiszka 2020-12-01 15:40 ` Henning Schild 0 siblings, 1 reply; 7+ messages in thread From: Jan Kiszka @ 2020-12-01 11:41 UTC (permalink / raw) To: [ext] Raphael Lisicki, isar-users On 01.12.20 11:41, [ext] Raphael Lisicki wrote: > Hi, > > the function do_adjust_git does not work if the fetcher has been set to > "gitsm" which is required if the project contains git submodules. > > See > https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26 > > > This will cause the .git/objects/info/alternates remain unmodified, > which in turn breaks git from inside the buildchroot. From https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher: "The Git Submodules fetcher is not a complete fetcher implementation. The fetcher has known issues where it does not use the normal source mirroring infrastructure properly. [...]" For that reason already, I would strongly discourage the use of this fetcher. And that's why it was never tested with Isar (to my best knowledge). In general, git submodules are hairy. And specifically in the context of distro (like Debian), their presence indicate that you rather want them as "Build-Depends" and properly packaged. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher 2020-12-01 11:41 ` Jan Kiszka @ 2020-12-01 15:40 ` Henning Schild 2020-12-01 16:44 ` Jan Kiszka 2020-12-02 17:10 ` Raphael Lisicki 0 siblings, 2 replies; 7+ messages in thread From: Henning Schild @ 2020-12-01 15:40 UTC (permalink / raw) To: [ext] Jan Kiszka; +Cc: [ext] Raphael Lisicki, isar-users Am Tue, 1 Dec 2020 12:41:08 +0100 schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>: > On 01.12.20 11:41, [ext] Raphael Lisicki wrote: > > Hi, > > > > the function do_adjust_git does not work if the fetcher has been > > set to "gitsm" which is required if the project contains git > > submodules. > > > > See > > https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26 > > > > > > This will cause the .git/objects/info/alternates remain unmodified, > > which in turn breaks git from inside the buildchroot. > > From > https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher: > > "The Git Submodules fetcher is not a complete fetcher implementation. > The fetcher has known issues where it does not use the normal source > mirroring infrastructure properly. [...]" > > For that reason already, I would strongly discourage the use of this > fetcher. And that's why it was never tested with Isar (to my best > knowledge). I have seen people use git submodules in isar recipes. But they are hairy indeed. A big issue is whether those submodules are absolute or relative to their main repo. Relative is your best bet, absolute will break authentication since you might find yourself wanting to clone with ssh and gitsm contains absolute https links. If you control that main repo, you might want to switch to relative ... if you can. Doing that breaks the "forking workflow" ... now every contributor needs to fork all, or switch to absolute before working. The worst case would be nested submodules ... I found it best to switch over to repeating the submodule stuff in the recipe, naming all modules with their shas and where they should be cloned. Hard to maintain ... but you can write helper scripts writing ".inc" files containing the bits and using a clone with initialized modules as input. SRC_URI = "gitsm://foo.git;destsuffix=${P}" SRCREV = "0815" becomes SRC_URI = "git://foo.git;destsuffix=${P};name=main SRCREV_main = "0815" SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1 SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2 SRCREV_module1 = "1234" SRCREV_module2 = "abcd" Those last 4 lines can be generated from ".gitmodules" and "git ls-tree <main-repo-sha> <sm-path>" and stored in an include. Henning > In general, git submodules are hairy. And specifically in the context > of distro (like Debian), their presence indicate that you rather want > them as "Build-Depends" and properly packaged. > > Jan > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher 2020-12-01 15:40 ` Henning Schild @ 2020-12-01 16:44 ` Jan Kiszka 2020-12-01 18:51 ` Henning Schild 2020-12-02 17:10 ` Raphael Lisicki 1 sibling, 1 reply; 7+ messages in thread From: Jan Kiszka @ 2020-12-01 16:44 UTC (permalink / raw) To: Henning Schild; +Cc: [ext] Raphael Lisicki, isar-users On 01.12.20 16:40, Henning Schild wrote: > Am Tue, 1 Dec 2020 12:41:08 +0100 > schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>: > >> On 01.12.20 11:41, [ext] Raphael Lisicki wrote: >>> Hi, >>> >>> the function do_adjust_git does not work if the fetcher has been >>> set to "gitsm" which is required if the project contains git >>> submodules. >>> >>> See >>> https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26 >>> >>> >>> This will cause the .git/objects/info/alternates remain unmodified, >>> which in turn breaks git from inside the buildchroot. >> >> From >> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher: >> >> "The Git Submodules fetcher is not a complete fetcher implementation. >> The fetcher has known issues where it does not use the normal source >> mirroring infrastructure properly. [...]" >> >> For that reason already, I would strongly discourage the use of this >> fetcher. And that's why it was never tested with Isar (to my best >> knowledge). > > I have seen people use git submodules in isar recipes. But they are > hairy indeed. A big issue is whether those submodules are absolute or > relative to their main repo. Relative is your best bet, absolute will > break authentication since you might find yourself wanting to clone with > ssh and gitsm contains absolute https links. > > If you control that main repo, you might want to switch to relative ... > if you can. Doing that breaks the "forking workflow" ... now every > contributor needs to fork all, or switch to absolute before working. > > The worst case would be nested submodules ... > > I found it best to switch over to repeating the submodule stuff in the > recipe, naming all modules with their shas and where they should be > cloned. Hard to maintain ... but you can write helper scripts writing > ".inc" files containing the bits and using a clone with initialized > modules as input. > > SRC_URI = "gitsm://foo.git;destsuffix=${P}" > SRCREV = "0815" > > becomes > > SRC_URI = "git://foo.git;destsuffix=${P};name=main > SRCREV_main = "0815" > SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1 > SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2 > SRCREV_module1 = "1234" > SRCREV_module2 = "abcd" > > Those last 4 lines can be generated from ".gitmodules" and "git ls-tree > <main-repo-sha> <sm-path>" and stored in an include. > That's a better pattern, indeed. Maybe someone wants to hack up some scripts/unroll-gitsm.sh to spit out those lines for a given recipe. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher 2020-12-01 16:44 ` Jan Kiszka @ 2020-12-01 18:51 ` Henning Schild 0 siblings, 0 replies; 7+ messages in thread From: Henning Schild @ 2020-12-01 18:51 UTC (permalink / raw) To: Jan Kiszka; +Cc: [ext] Raphael Lisicki, isar-users Am Tue, 1 Dec 2020 17:44:56 +0100 schrieb Jan Kiszka <jan.kiszka@siemens.com>: > On 01.12.20 16:40, Henning Schild wrote: > > Am Tue, 1 Dec 2020 12:41:08 +0100 > > schrieb "[ext] Jan Kiszka" <jan.kiszka@siemens.com>: > > > >> On 01.12.20 11:41, [ext] Raphael Lisicki wrote: > >>> Hi, > >>> > >>> the function do_adjust_git does not work if the fetcher has been > >>> set to "gitsm" which is required if the project contains git > >>> submodules. > >>> > >>> See > >>> https://github.com/siemens/isar/blob/master/meta/classes/dpkg-base.bbclass#L26 > >>> > >>> > >>> This will cause the .git/objects/info/alternates remain > >>> unmodified, which in turn breaks git from inside the buildchroot. > >>> > >> > >> From > >> https://www.yoctoproject.org/docs/latest/bitbake-user-manual/bitbake-user-manual.html#gitsm-fetcher: > >> > >> "The Git Submodules fetcher is not a complete fetcher > >> implementation. The fetcher has known issues where it does not use > >> the normal source mirroring infrastructure properly. [...]" > >> > >> For that reason already, I would strongly discourage the use of > >> this fetcher. And that's why it was never tested with Isar (to my > >> best knowledge). > > > > I have seen people use git submodules in isar recipes. But they are > > hairy indeed. A big issue is whether those submodules are absolute > > or relative to their main repo. Relative is your best bet, absolute > > will break authentication since you might find yourself wanting to > > clone with ssh and gitsm contains absolute https links. > > > > If you control that main repo, you might want to switch to relative > > ... if you can. Doing that breaks the "forking workflow" ... now > > every contributor needs to fork all, or switch to absolute before > > working. > > > > The worst case would be nested submodules ... > > > > I found it best to switch over to repeating the submodule stuff in > > the recipe, naming all modules with their shas and where they > > should be cloned. Hard to maintain ... but you can write helper > > scripts writing ".inc" files containing the bits and using a clone > > with initialized modules as input. > > > > SRC_URI = "gitsm://foo.git;destsuffix=${P}" > > SRCREV = "0815" > > > > becomes > > > > SRC_URI = "git://foo.git;destsuffix=${P};name=main > > SRCREV_main = "0815" > > SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1 > > SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2 > > SRCREV_module1 = "1234" > > SRCREV_module2 = "abcd" > > > > Those last 4 lines can be generated from ".gitmodules" and "git > > ls-tree <main-repo-sha> <sm-path>" and stored in an include. > > > > That's a better pattern, indeed. > > Maybe someone wants to hack up some scripts/unroll-gitsm.sh to spit > out those lines for a given recipe. Coming up with a truly generic script for that would be like fixing the gitsm fetcher, pretty hard. And if anyone wants to go ahead, fixing the sm fetcher in bitbake would be better. Writing a script for a single recipe is "easy", so that is maybe what people should start with. The real problem is getting getting the url right, it can not be taken from ".gitmodules". Because there you get the ssh vs http(s). Absolute vs. relative ... in the worst case with PREMIRROS. And if you are really unlucky with submodule recursion. I currently work on a layer where we have scripts generating includes to bump recipes. Those scripts are per recipe and very layer-specific. There it is mainly about generating the checksums that bitbake wants in the recipes. Bitbake could learn from gentoo, but that is another story. Henning > Jan > ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher 2020-12-01 15:40 ` Henning Schild 2020-12-01 16:44 ` Jan Kiszka @ 2020-12-02 17:10 ` Raphael Lisicki 2020-12-02 18:03 ` Jan Kiszka 1 sibling, 1 reply; 7+ messages in thread From: Raphael Lisicki @ 2020-12-02 17:10 UTC (permalink / raw) To: isar-users; +Cc: Henning Schild, [ext] Jan Kiszka Thanks for your answers. I know well about the pains of the submodules. > > SRC_URI = "gitsm://foo.git;destsuffix=${P}" > SRCREV = "0815" > > becomes > > SRC_URI = "git://foo.git;destsuffix=${P};name=main > SRCREV_main = "0815" > SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1 > SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2 > SRCREV_module1 = "1234" > SRCREV_module2 = "abcd" > This sounds like a feasible workaround. Happily, I was able to just switch to the git fetcher. Simply adding "and ud.type != 'gitsm'" would also fix this for certain cases (mine :D ) . Given that it is known that gitsm is not working properly, such a best-effort fix might also be an option. raphael ^ permalink raw reply [flat|nested] 7+ messages in thread
* Re: do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher 2020-12-02 17:10 ` Raphael Lisicki @ 2020-12-02 18:03 ` Jan Kiszka 0 siblings, 0 replies; 7+ messages in thread From: Jan Kiszka @ 2020-12-02 18:03 UTC (permalink / raw) To: Raphael Lisicki, isar-users; +Cc: Henning Schild On 02.12.20 18:10, Raphael Lisicki wrote: > Thanks for your answers. I know well about the pains of the submodules. > >> >> SRC_URI = "gitsm://foo.git;destsuffix=${P}" >> SRCREV = "0815" >> >> becomes >> >> SRC_URI = "git://foo.git;destsuffix=${P};name=main >> SRCREV_main = "0815" >> SRC_URI += "git://bar.git;destsuffix=${P}/modules/bar;name=module1 >> SRC_URI += "git://baz.git;destsuffix=${P}/mod/baz;name=module2 >> SRCREV_module1 = "1234" >> SRCREV_module2 = "abcd" >> > > This sounds like a feasible workaround. Happily, I was able to just > switch to the git fetcher. > > Simply adding "and ud.type != 'gitsm'" would also fix this for certain > cases (mine :D ) . Given that it is known that gitsm is not working > properly, such a best-effort fix might also be an option. > Feel free to send such a trivial patch. If it makes things a bit better, no need to not merge that. Jan -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 7+ messages in thread
end of thread, other threads:[~2020-12-02 18:03 UTC | newest] Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2020-12-01 10:41 do_adjust_git in dpkg-base.bbclass not working for gitsm fetcher Raphael Lisicki 2020-12-01 11:41 ` Jan Kiszka 2020-12-01 15:40 ` Henning Schild 2020-12-01 16:44 ` Jan Kiszka 2020-12-01 18:51 ` Henning Schild 2020-12-02 17:10 ` Raphael Lisicki 2020-12-02 18:03 ` Jan Kiszka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox