* [PATCH 0/3] Make adjust_git work in both worlds
@ 2021-08-12 9:23 Jan Kiszka
2021-08-12 9:23 ` [PATCH 1/3] dpkg-base: Lock do_adjust_git against each other Jan Kiszka
` (2 more replies)
0 siblings, 3 replies; 5+ messages in thread
From: Jan Kiszka @ 2021-08-12 9:23 UTC (permalink / raw)
To: isar-users; +Cc: Uladzimir Bely, Henning Schild
Finally allow the adjusted git repos work both inside and outside the
buildchroots. Also fixes the race introduced by patching links in
DL_DIR.
NOTE: Not yet fully tested, our CI is busy with that now but just
stumbled again over the broken umount patches.
Jan
Jan Kiszka (3):
dpkg-base: Lock do_adjust_git against each other
dpkg-base: Make mirror link relative
Rework do_adjust_git to support inside and outside usage
meta/classes/dpkg-base.bbclass | 15 ++++++++++++---
meta/conf/bitbake.conf | 1 +
meta/recipes-devtools/buildchroot/buildchroot.inc | 2 ++
3 files changed, 15 insertions(+), 3 deletions(-)
--
2.31.1
^ permalink raw reply [flat|nested] 5+ messages in thread* [PATCH 1/3] dpkg-base: Lock do_adjust_git against each other 2021-08-12 9:23 [PATCH 0/3] Make adjust_git work in both worlds Jan Kiszka @ 2021-08-12 9:23 ` Jan Kiszka 2021-08-12 9:23 ` [PATCH 2/3] dpkg-base: Make mirror link relative Jan Kiszka 2021-08-12 9:23 ` [PATCH 3/3] Rework do_adjust_git to support inside and outside usage Jan Kiszka 2 siblings, 0 replies; 5+ messages in thread From: Jan Kiszka @ 2021-08-12 9:23 UTC (permalink / raw) To: isar-users; +Cc: Uladzimir Bely, Henning Schild From: Jan Kiszka <jan.kiszka@siemens.com> This is needed since e47ada143469 because we now adjust the links to mirrors in DL_DIR which may collide with other tasks doing this at the same time. Locking with also be needed soon for establishing the host-side link to the download dir. Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> --- meta/classes/dpkg-base.bbclass | 1 + 1 file changed, 1 insertion(+) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index ec8fbc1..fe16846 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -55,6 +55,7 @@ python do_adjust_git() { } addtask adjust_git before do_dpkg_build +do_adjust_git[lockfiles] += "${DL_DIR}/isar.lock" inherit patch addtask patch before do_adjust_git -- 2.31.1 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 2/3] dpkg-base: Make mirror link relative 2021-08-12 9:23 [PATCH 0/3] Make adjust_git work in both worlds Jan Kiszka 2021-08-12 9:23 ` [PATCH 1/3] dpkg-base: Lock do_adjust_git against each other Jan Kiszka @ 2021-08-12 9:23 ` Jan Kiszka 2021-08-12 9:23 ` [PATCH 3/3] Rework do_adjust_git to support inside and outside usage Jan Kiszka 2 siblings, 0 replies; 5+ messages in thread From: Jan Kiszka @ 2021-08-12 9:23 UTC (permalink / raw) To: isar-users; +Cc: Uladzimir Bely, Henning Schild From: Jan Kiszka <jan.kiszka@siemens.com> This will allow to use it both from the outer builder context and inside the buildchroot. No idea why bitbake doesn't to that in the first place. Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> --- meta/classes/dpkg-base.bbclass | 6 ++++-- 1 file changed, 4 insertions(+), 2 deletions(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index fe16846..a4c8663 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -28,8 +28,10 @@ python do_adjust_git() { if os.path.islink(ud.localpath): realpath = os.path.realpath(ud.localpath) - if realpath.startswith(d.getVar("DL_DIR")): - link = realpath.replace(d.getVar("DL_DIR"), '/downloads', 1) + filter_out = os.path.join(d.getVar("DL_DIR"), "git") + "/" + if realpath.startswith(filter_out): + # make the link relative + link = realpath.replace(filter_out, '', 1) os.unlink(ud.localpath) os.symlink(link, ud.localpath) -- 2.31.1 ^ permalink raw reply [flat|nested] 5+ messages in thread
* [PATCH 3/3] Rework do_adjust_git to support inside and outside usage 2021-08-12 9:23 [PATCH 0/3] Make adjust_git work in both worlds Jan Kiszka 2021-08-12 9:23 ` [PATCH 1/3] dpkg-base: Lock do_adjust_git against each other Jan Kiszka 2021-08-12 9:23 ` [PATCH 2/3] dpkg-base: Make mirror link relative Jan Kiszka @ 2021-08-12 9:23 ` Jan Kiszka 2021-08-12 10:31 ` Jan Kiszka 2 siblings, 1 reply; 5+ messages in thread From: Jan Kiszka @ 2021-08-12 9:23 UTC (permalink / raw) To: isar-users; +Cc: Uladzimir Bely, Henning Schild From: Jan Kiszka <jan.kiszka@siemens.com> Rewrite the alternates entry to use a relative path outside of the workdir. That linke can then be created independently of the outer build environment as well as inside the buildchroot. We use ${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/.git-downloads outside and /home/.git-downloads inside the buildchroot. Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> --- meta/classes/dpkg-base.bbclass | 8 +++++++- meta/conf/bitbake.conf | 1 + meta/recipes-devtools/buildchroot/buildchroot.inc | 2 ++ 3 files changed, 10 insertions(+), 1 deletion(-) diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass index a4c8663..cd6c649 100644 --- a/meta/classes/dpkg-base.bbclass +++ b/meta/classes/dpkg-base.bbclass @@ -19,6 +19,12 @@ python do_adjust_git() { rootdir = d.getVar('WORKDIR', True) + git_link = os.path.join(d.getVar('GIT_DL_LINK_DIR'), '.git-downloads') + git_dl = os.path.join(d.getVar("DL_DIR"), "git") + + if not os.path.exists(git_link) or os.path.realpath(git_link) != git_dl: + os.symlink(git_dl, git_link) + for src_uri in (d.getVar("SRC_URI", True) or "").split(): try: fetcher = bb.fetch2.Fetch([src_uri], d) @@ -48,7 +54,7 @@ python do_adjust_git() { if os.path.exists(alternates): cmd = ["sed", "-i", alternates, "-e", - "s|{}|/downloads|".format(d.getVar("DL_DIR"))] + "s|{}|../../../../../.git-downloads|".format(git_dl)] bb.note(' '.join(cmd)) if subprocess.call(cmd) != 0: bb.fatal("git alternates adjustment failed") diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf index 9859456..7f5901d 100644 --- a/meta/conf/bitbake.conf +++ b/meta/conf/bitbake.conf @@ -48,6 +48,7 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}" # isar specific config WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}" +GIT_DL_LINK_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}" DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap" DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot" DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot" diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc index 0c1fb7f..52a5f1c 100644 --- a/meta/recipes-devtools/buildchroot/buildchroot.inc +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc @@ -67,6 +67,8 @@ buildchroot_install_files() { sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/ sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/ + sudo ln -s /downloads/git "${BUILDCHROOT_DIR}/home/.git-downloads" + # Configure root filesystem sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR} USER_ID=$(id -u) -- 2.31.1 ^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [PATCH 3/3] Rework do_adjust_git to support inside and outside usage 2021-08-12 9:23 ` [PATCH 3/3] Rework do_adjust_git to support inside and outside usage Jan Kiszka @ 2021-08-12 10:31 ` Jan Kiszka 0 siblings, 0 replies; 5+ messages in thread From: Jan Kiszka @ 2021-08-12 10:31 UTC (permalink / raw) To: isar-users; +Cc: Uladzimir Bely, Henning Schild On 12.08.21 11:23, Jan Kiszka wrote: > From: Jan Kiszka <jan.kiszka@siemens.com> > > Rewrite the alternates entry to use a relative path outside of the > workdir. That linke can then be created independently of the outer build > environment as well as inside the buildchroot. > > We use ${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/.git-downloads outside > and /home/.git-downloads inside the buildchroot. > > Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com> > --- > meta/classes/dpkg-base.bbclass | 8 +++++++- > meta/conf/bitbake.conf | 1 + > meta/recipes-devtools/buildchroot/buildchroot.inc | 2 ++ > 3 files changed, 10 insertions(+), 1 deletion(-) > > diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass > index a4c8663..cd6c649 100644 > --- a/meta/classes/dpkg-base.bbclass > +++ b/meta/classes/dpkg-base.bbclass > @@ -19,6 +19,12 @@ python do_adjust_git() { > > rootdir = d.getVar('WORKDIR', True) > > + git_link = os.path.join(d.getVar('GIT_DL_LINK_DIR'), '.git-downloads') > + git_dl = os.path.join(d.getVar("DL_DIR"), "git") > + > + if not os.path.exists(git_link) or os.path.realpath(git_link) != git_dl: > + os.symlink(git_dl, git_link) > + > for src_uri in (d.getVar("SRC_URI", True) or "").split(): > try: > fetcher = bb.fetch2.Fetch([src_uri], d) > @@ -48,7 +54,7 @@ python do_adjust_git() { > > if os.path.exists(alternates): > cmd = ["sed", "-i", alternates, "-e", > - "s|{}|/downloads|".format(d.getVar("DL_DIR"))] > + "s|{}|../../../../../.git-downloads|".format(git_dl)] Hmm, this does not yet account for the case that the repo is placed in some subdir under workdir. Will make this more general. I also forgot to fix the task dependencies - devshell is not pulling adjust_git. Jan > bb.note(' '.join(cmd)) > if subprocess.call(cmd) != 0: > bb.fatal("git alternates adjustment failed") > diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf > index 9859456..7f5901d 100644 > --- a/meta/conf/bitbake.conf > +++ b/meta/conf/bitbake.conf > @@ -48,6 +48,7 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}" > > # isar specific config > WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}" > +GIT_DL_LINK_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}" > DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap" > DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot" > DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot" > diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc > index 0c1fb7f..52a5f1c 100644 > --- a/meta/recipes-devtools/buildchroot/buildchroot.inc > +++ b/meta/recipes-devtools/buildchroot/buildchroot.inc > @@ -67,6 +67,8 @@ buildchroot_install_files() { > sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/ > sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/ > > + sudo ln -s /downloads/git "${BUILDCHROOT_DIR}/home/.git-downloads" > + > # Configure root filesystem > sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR} > USER_ID=$(id -u) > -- Siemens AG, T RDA IOT Corporate Competence Center Embedded Linux ^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2021-08-12 10:31 UTC | newest] Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2021-08-12 9:23 [PATCH 0/3] Make adjust_git work in both worlds Jan Kiszka 2021-08-12 9:23 ` [PATCH 1/3] dpkg-base: Lock do_adjust_git against each other Jan Kiszka 2021-08-12 9:23 ` [PATCH 2/3] dpkg-base: Make mirror link relative Jan Kiszka 2021-08-12 9:23 ` [PATCH 3/3] Rework do_adjust_git to support inside and outside usage Jan Kiszka 2021-08-12 10:31 ` Jan Kiszka
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox