[PATCH v3 0/4] Make adjust_git work in both worlds

[PATCH v3 0/4] Make adjust_git work in both worlds

[Jan Kiszka]

Finally allow the adjusted git repos work both inside and outside the
buildchroots. Also fixes the race introduced by patching links in
DL_DIR.

Changes in v3:
 - use /git for lock
 - fix typos in commit logs

Changes in v2:
 - build relative alternates path so that destsuffix paths are respected
 - fix task dependencies so that devshell gets adjusted git again

Jan

Jan Kiszka (4):
  dpkg-base: Lock do_adjust_git against each other
  dpkg-base: Make mirror link relative
  Rework do_adjust_git to support inside and outside usage
  Revert "dpkg: adjust task order to allow using "git" for patching"

 meta/classes/dpkg-base.bbclass                | 22 ++++++++++++++-----
 meta/conf/bitbake.conf                        |  1 +
 .../buildchroot/buildchroot.inc               |  2 ++
 3 files changed, 20 insertions(+), 5 deletions(-)

-- 
2.31.1

[PATCH v3 1/4] dpkg-base: Lock do_adjust_git against each other

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This is needed since e47ada143469 because we now adjust the links to
mirrors in DL_DIR/git which may collide with other tasks doing this at
the same time.

Locking will also be needed soon for establishing the host-side link
to the download dir.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ec8fbc1..2cff1b3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -55,6 +55,7 @@ python do_adjust_git() {
 }
 
 addtask adjust_git before do_dpkg_build
+do_adjust_git[lockfiles] += "${DL_DIR}/git/isar.lock"
 
 inherit patch
 addtask patch before do_adjust_git
-- 
2.31.1

[PATCH v3 2/4] dpkg-base: Make mirror link relative

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This will allow to use it both from the outer builder context and inside
the buildchroot. No idea why bitbake doesn't do that in the first place.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 6 ++++--
 1 file changed, 4 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 2cff1b3..ca1dccf 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -28,8 +28,10 @@ python do_adjust_git() {
 
             if os.path.islink(ud.localpath):
                 realpath = os.path.realpath(ud.localpath)
-                if realpath.startswith(d.getVar("DL_DIR")):
-                    link = realpath.replace(d.getVar("DL_DIR"), '/downloads', 1)
+                filter_out = os.path.join(d.getVar("DL_DIR"), "git") + "/"
+                if realpath.startswith(filter_out):
+                    # make the link relative
+                    link = realpath.replace(filter_out, '', 1)
                     os.unlink(ud.localpath)
                     os.symlink(link, ud.localpath)
 
-- 
2.31.1

[PATCH v3 3/4] Rework do_adjust_git to support inside and outside usage

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

Rewrite the alternates entry to use a relative path outside of the
workdir. That linke can then be created independently of the outer build
environment as well as inside the buildchroot.

We use ${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/.git-downloads outside
and /home/.git-downloads inside the buildchroot.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass                    | 11 ++++++++++-
 meta/conf/bitbake.conf                            |  1 +
 meta/recipes-devtools/buildchroot/buildchroot.inc |  2 ++
 3 files changed, 13 insertions(+), 1 deletion(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ca1dccf..9a18acb 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -19,6 +19,12 @@ python do_adjust_git() {
 
     rootdir = d.getVar('WORKDIR', True)
 
+    git_link = os.path.join(d.getVar('GIT_DL_LINK_DIR'), '.git-downloads')
+    git_dl = os.path.join(d.getVar("DL_DIR"), "git")
+
+    if not os.path.exists(git_link) or os.path.realpath(git_link) != git_dl:
+        os.symlink(git_dl, git_link)
+
     for src_uri in (d.getVar("SRC_URI", True) or "").split():
         try:
             fetcher = bb.fetch2.Fetch([src_uri], d)
@@ -44,11 +50,14 @@ python do_adjust_git() {
             destsuffix = ud.parm.get("destsuffix", def_destsuffix)
             destdir = ud.destdir = os.path.join(rootdir, destsuffix)
 
+            git_link_rel = os.path.relpath(git_link,
+                                           os.path.join(destdir, ".git/objects"))
+
             alternates = os.path.join(destdir, ".git/objects/info/alternates")
 
             if os.path.exists(alternates):
                 cmd = ["sed", "-i", alternates, "-e",
-                       "s|{}|/downloads|".format(d.getVar("DL_DIR"))]
+                       "s|{}|{}|".format(git_dl, git_link_rel)]
                 bb.note(' '.join(cmd))
                 if subprocess.call(cmd) != 0:
                     bb.fatal("git alternates adjustment failed")
diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 9859456..7f5901d 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -48,6 +48,7 @@ GITPKGV = "${@bb.fetch2.get_srcrev(d, 'gitpkgv_revision')}"
 
 # isar specific config
 WORKDIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}/${PN}/${PV}-${PR}"
+GIT_DL_LINK_DIR = "${TMPDIR}/work/${DISTRO}-${DISTRO_ARCH}"
 DEPLOY_DIR_BOOTSTRAP = "${DEPLOY_DIR}/bootstrap"
 DEPLOY_DIR_BUILDCHROOT = "${DEPLOY_DIR}/buildchroot"
 DEPLOY_DIR_SDKCHROOT = "${DEPLOY_DIR}/sdkchroot"
diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 0c1fb7f..52a5f1c 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -67,6 +67,8 @@ buildchroot_install_files() {
     sudo install -m 755 ${WORKDIR}/common.sh ${BUILDCHROOT_DIR}/isar/
     sudo install -m 755 ${WORKDIR}/deps.sh ${BUILDCHROOT_DIR}/isar/
 
+    sudo ln -s /downloads/git "${BUILDCHROOT_DIR}/home/.git-downloads"
+
     # Configure root filesystem
     sudo install -m 755 ${WORKDIR}/configscript.sh ${BUILDCHROOT_DIR}
     USER_ID=$(id -u)
-- 
2.31.1

[PATCH v3 4/4] Revert "dpkg: adjust task order to allow using "git" for patching"

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This reverts commit b4c33ba69ed89849ecf32d8731aa97ecf88226f5.

No longer needed now that adjust_git stopped breaking the outer
environment. Moreover, this was broken /wrt devshell anyway.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---
 meta/classes/dpkg-base.bbclass | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 9a18acb..8286168 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -65,11 +65,11 @@ python do_adjust_git() {
             bb.fatal(str(e))
 }
 
-addtask adjust_git before do_dpkg_build
+addtask adjust_git after do_unpack before do_patch
 do_adjust_git[lockfiles] += "${DL_DIR}/git/isar.lock"
 
 inherit patch
-addtask patch before do_adjust_git
+addtask patch after do_adjust_git before do_dpkg_build
 
 SRC_APT ?= ""
 
-- 
2.31.1

Re: [PATCH v3 0/4] Make adjust_git work in both worlds

[Anton Mikanovich]

26.08.2021 12:22, Jan Kiszka wrote:
> Finally allow the adjusted git repos work both inside and outside the
> buildchroots. Also fixes the race introduced by patching links in
> DL_DIR.
>
> Changes in v3:
>   - use /git for lock
>   - fix typos in commit logs
>
> Changes in v2:
>   - build relative alternates path so that destsuffix paths are respected
>   - fix task dependencies so that devshell gets adjusted git again
>
> Jan
>
> Jan Kiszka (4):
>    dpkg-base: Lock do_adjust_git against each other
>    dpkg-base: Make mirror link relative
>    Rework do_adjust_git to support inside and outside usage
>    Revert "dpkg: adjust task order to allow using "git" for patching"
>
>   meta/classes/dpkg-base.bbclass                | 22 ++++++++++++++-----
>   meta/conf/bitbake.conf                        |  1 +
>   .../buildchroot/buildchroot.inc               |  2 ++
>   3 files changed, 20 insertions(+), 5 deletions(-)
>
Applied to next, thanks.

-- 
Anton Mikanovich
Promwad Ltd.
External service provider of ilbers GmbH
Maria-Merian-Str. 8
85521 Ottobrunn, Germany
+49 (89) 122 67 24-0
Commercial register Munich, HRB 214197
General Manager: Baurzhan Ismagulov

Re: [PATCH v3 1/4] dpkg-base: Lock do_adjust_git against each other

[Jan Kiszka]

On 26.08.21 10:00, Jan Kiszka wrote:
> From: Jan Kiszka <jan.kiszka@siemens.com>
> 
> This is needed since e47ada143469 because we now adjust the links to
> mirrors in DL_DIR/git which may collide with other tasks doing this at
> the same time.
> 
> Locking will also be needed soon for establishing the host-side link
> to the download dir.
> 
> Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
> ---
> 
> Changes in v3:
>  - move lock into git folder
>  - fix commit message
> 
>  meta/classes/dpkg-base.bbclass | 1 +
>  1 file changed, 1 insertion(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index ec8fbc1..2cff1b3 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -55,6 +55,7 @@ python do_adjust_git() {
>  }
>  
>  addtask adjust_git before do_dpkg_build
> +do_adjust_git[lockfiles] += "${DL_DIR}/git/isar.lock"
>  
>  inherit patch
>  addtask patch before do_adjust_git
> 

Hmpf, causes collisions further downstream, will resend whole series.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[PATCH v3 1/4] dpkg-base: Lock do_adjust_git against each other

[Jan Kiszka]

From: Jan Kiszka <jan.kiszka@siemens.com>

This is needed since e47ada143469 because we now adjust the links to
mirrors in DL_DIR/git which may collide with other tasks doing this at
the same time.

Locking will also be needed soon for establishing the host-side link
to the download dir.

Signed-off-by: Jan Kiszka <jan.kiszka@siemens.com>
---

Changes in v3:
 - move lock into git folder
 - fix commit message

 meta/classes/dpkg-base.bbclass | 1 +
 1 file changed, 1 insertion(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index ec8fbc1..2cff1b3 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -55,6 +55,7 @@ python do_adjust_git() {
 }
 
 addtask adjust_git before do_dpkg_build
+do_adjust_git[lockfiles] += "${DL_DIR}/git/isar.lock"
 
 inherit patch
 addtask patch before do_adjust_git
-- 
2.31.1