[PATCH v1 0/2] Implement salsa.debian.org packages mirroring

[PATCH v1 0/2] Implement salsa.debian.org packages mirroring

[Anton Mikanovich]

We've already experienced build issues caused by 503 Service Unavailable
on https://salsa.debian.org which is used as SRC_URI of cowsay and
libubootenv packages. To have a backup of package sources we need to use
MIRRORS bitbake variable. But Isar building fails because absolute
symlink to mirror directory created by bitbake is not exists inside
chroot. We need to adjust those links before do_dpkg_build stage.

Working on upstream fix in bitbake is already started, but it will not
be done quickly because easy fix is not working:
https://lists.openembedded.org/g/bitbake-devel/topic/78342988

Anton Mikanovich (2):
  dpkg: Adjust symlinks for mirrors
  conf: Add ilbers mirror support for salsa repos

 meta-isar/conf/local.conf.sample | 3 +++
 meta/classes/dpkg-base.bbclass   | 7 +++++++
 2 files changed, 10 insertions(+)

-- 
2.25.1

[PATCH v1 1/2] dpkg: Adjust symlinks for mirrors

[Anton Mikanovich]

When bitbake fetch git repository from mirror it creates symlink from
original repository name. This symlink use absolute path, which is not
exist inside chroot when building the package. So we need to adjust this
path just like we do for git alternates path.

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta/classes/dpkg-base.bbclass | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
index 5c7bddc..23dc191 100644
--- a/meta/classes/dpkg-base.bbclass
+++ b/meta/classes/dpkg-base.bbclass
@@ -26,6 +26,13 @@ python do_adjust_git() {
             if ud.type != 'git':
                 continue
 
+            if os.path.islink(ud.localpath):
+                realpath = os.path.realpath(ud.localpath)
+                if realpath.startswith(d.getVar("DL_DIR")):
+                    link = realpath.replace(d.getVar("DL_DIR"), '/downloads', 1)
+                    os.unlink(ud.localpath)
+                    os.symlink(link, ud.localpath)
+
             subdir = ud.parm.get("subpath", "")
             if subdir != "":
                 def_destsuffix = "%s/" % os.path.basename(subdir.rstrip('/'))
-- 
2.25.1

Re: [PATCH v1 1/2] dpkg: Adjust symlinks for mirrors

[Jan Kiszka]

On 23.03.21 12:50, Anton Mikanovich wrote:
> When bitbake fetch git repository from mirror it creates symlink from
> original repository name. This symlink use absolute path, which is not
> exist inside chroot when building the package. So we need to adjust this
> path just like we do for git alternates path.
> 
> Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
> ---
>  meta/classes/dpkg-base.bbclass | 7 +++++++
>  1 file changed, 7 insertions(+)
> 
> diff --git a/meta/classes/dpkg-base.bbclass b/meta/classes/dpkg-base.bbclass
> index 5c7bddc..23dc191 100644
> --- a/meta/classes/dpkg-base.bbclass
> +++ b/meta/classes/dpkg-base.bbclass
> @@ -26,6 +26,13 @@ python do_adjust_git() {
>              if ud.type != 'git':
>                  continue
>  
> +            if os.path.islink(ud.localpath):
> +                realpath = os.path.realpath(ud.localpath)
> +                if realpath.startswith(d.getVar("DL_DIR")):
> +                    link = realpath.replace(d.getVar("DL_DIR"), '/downloads', 1)
> +                    os.unlink(ud.localpath)
> +                    os.symlink(link, ud.localpath)
> +
>              subdir = ud.parm.get("subpath", "")
>              if subdir != "":
>                  def_destsuffix = "%s/" % os.path.basename(subdir.rstrip('/'))
> 

This is racy and can generate random build breakages if a git repo
should be used by multiple recipes. I'll fix this up by locking all
adjust_git tasks, which I need for making it compatible it compatible
with git-patching anyway.

Jan

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux

[PATCH v1 2/2] conf: Add ilbers mirror support for salsa repos

[Anton Mikanovich]

Signed-off-by: Anton Mikanovich <amikan@ilbers.de>
---
 meta-isar/conf/local.conf.sample | 3 +++
 1 file changed, 3 insertions(+)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 107496c..4ab8a27 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -147,6 +147,9 @@ BB_DISKMON_DIRS = "\
     ABORT,${SSTATE_DIR},100M,1K \
     ABORT,/tmp,10M,1K"
 
+# Isar MIRRORS in case of service unavailable
+MIRRORS ?= "git?://salsa\.debian\.org/debian/.* git://github.com/ilbers/BASENAME"
+
 #
 # Shared-state files from other locations
 #
-- 
2.25.1