[RFC PATCH 0/2] Buildchroot non-determinism

[RFC PATCH 0/2] Buildchroot non-determinism

[Vijai Kumar K]

Hello All,

Sometimes buildchroot-host completes long before isar-bootstrap-target
and proceeds with building packages and populates it in isar-apt.

By the time buildchroot-target is triggered, some packages are available
via isar-apt and as per the default preference[1], is installed instead
of the one from other mirrors.

This is not always the case, sometimes there is no such race and the
packages from DISTRO_APT_SOURCES mirrors are installed.

This makes buildchroot non-deterministic and affects caching.

One solution is to provide a mechanism to set preferences for
buildchroot. This way, the user would have flexibility to control the
mirrors from which certain packages are installed.

This RFC introduces 2 new variables
1. BUILDCHROOT_HOST_APT_PREFERENCES
2. BUILDCHROOT_TARGET_APT_PREFERENCES
to set the preferences for buildchroot-host/target respectively.

Ofcourse, there might be more ways to solve this problem. Happy to
discuss them here.

Thanks,
Vijai Kumar K

[1]: https://github.com/ilbers/isar/blob/919fc995fc8ecb675f8bf639ee8628f90728b1ea/meta/classes/rootfs.bbclass#L90

Vijai Kumar K (2):
  meta: Move aggregate_files to base class
  buildchroot: Add provision to specify apt preferences

 meta/classes/base.bbclass                     | 10 +++++++
 .../isar-bootstrap/isar-bootstrap.inc         | 10 -------
 .../buildchroot/buildchroot.inc               | 30 +++++++++++++++++++
 3 files changed, 40 insertions(+), 10 deletions(-)

-- 
2.25.1

[RFC PATCH 1/2] meta: Move aggregate_files to base class

[Vijai Kumar K]

This API is generic and can be used to aggregate any set of files.
This can be reused for building buildchroot preferences file in future.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 meta/classes/base.bbclass                           | 10 ++++++++++
 meta/recipes-core/isar-bootstrap/isar-bootstrap.inc | 10 ----------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/meta/classes/base.bbclass b/meta/classes/base.bbclass
index 72d4cc0..6d24ca7 100644
--- a/meta/classes/base.bbclass
+++ b/meta/classes/base.bbclass
@@ -235,3 +235,13 @@ def base_set_filespath(path, d):
             if p != "":
                 filespath.append(os.path.join(p, o))
     return ":".join(filespath)
+
+def aggregate_files(d, file_list, file_out):
+    import shutil
+
+    with open(file_out, "wb") as out_fd:
+        for entry in file_list:
+            entry_real = bb.parse.resolve_file(entry, d)
+            with open(entry_real, "rb") as in_fd:
+                 shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
+            out_fd.write("\n".encode())
diff --git a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
index e9f9291..bc8ce22 100644
--- a/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
+++ b/meta/recipes-core/isar-bootstrap/isar-bootstrap.inc
@@ -68,16 +68,6 @@ python () {
         d.appendVar("SRC_URI", " file://%s" % bb.parse.resolve_file(file, d))
 }
 
-def aggregate_files(d, file_list, file_out):
-    import shutil
-
-    with open(file_out, "wb") as out_fd:
-        for entry in file_list:
-            entry_real = bb.parse.resolve_file(entry, d)
-            with open(entry_real, "rb") as in_fd:
-                 shutil.copyfileobj(in_fd, out_fd, 1024*1024*10)
-            out_fd.write("\n".encode())
-
 def parse_aptsources_list_line(source_list_line):
     import re
 
-- 
2.25.1

[RFC PATCH 2/2] buildchroot: Add provision to specify apt preferences

[Vijai Kumar K]

Add provision to set apt preferences for buildchroot-host and target.

Signed-off-by: Vijai Kumar K <Vijaikumar_Kanagarajan@mentor.com>
---
 .../buildchroot/buildchroot.inc               | 30 +++++++++++++++++++
 1 file changed, 30 insertions(+)

diff --git a/meta/recipes-devtools/buildchroot/buildchroot.inc b/meta/recipes-devtools/buildchroot/buildchroot.inc
index 726c7bb..fd06b80 100644
--- a/meta/recipes-devtools/buildchroot/buildchroot.inc
+++ b/meta/recipes-devtools/buildchroot/buildchroot.inc
@@ -50,6 +50,36 @@ rootfs_do_mounts_append() {
 EOSUDO
 }
 
+BUILDCHROOT_VARS_PREFIX ?= "${@'BUILDCHROOT_HOST_' if d.getVar('BOOTSTRAP_VARIANT') == 'HOST' else 'BUILDCHROOT_TARGET_'}"
+python() {
+    distro_apt_preferences = d.getVar(d.getVar("BUILDCHROOT_VARS_PREFIX") + "APT_PREFERENCES", True) or ""
+    for file in distro_apt_preferences.split():
+        d.appendVar("SRC_URI", " file://%s" % bb.parse.resolve_file(file, d))
+}
+
+APTPREFS = "${WORKDIR}/apt-preferences"
+
+do_buildchroot_prepare[vardeps] += " \
+    APTPREFS \
+    ${BUILDCHROOT_VARS_PREFIX}_APT_PREFERENCES \
+    "
+
+python do_buildchroot_prepare() {
+    apt_preferences_out = d.getVar("APTPREFS", True)
+    apt_preferences_list = (
+        d.getVar(d.getVar("BUILDCHROOT_VARS_PREFIX") + "APT_PREFERENCES", True) or ""
+    ).split()
+    aggregate_files(d, apt_preferences_list, apt_preferences_out)
+}
+addtask buildchroot_prepare before do_rootfs_install after do_unpack
+
+ROOTFS_CONFIGURE_COMMAND += "buildchroot_configure_preferences"
+buildchroot_configure_preferences() {
+    sudo install -v -m644 "${APTPREFS}" \
+                         "${BUILDCHROOT_DIR}/etc/apt/preferences.d/buildchroot"
+}
+
+
 ROOTFS_POSTPROCESS_COMMAND =+ "buildchroot_install_files"
 buildchroot_install_files() {
     sudo mkdir -p "${BUILDCHROOT_DIR}/home/builder"
-- 
2.25.1

Re: [RFC PATCH 0/2] Buildchroot non-determinism

[Jan Kiszka]

On 29.11.21 15:52, Vijai Kumar K wrote:
> Hello All,
> 
> Sometimes buildchroot-host completes long before isar-bootstrap-target
> and proceeds with building packages and populates it in isar-apt.
> 
> By the time buildchroot-target is triggered, some packages are available
> via isar-apt and as per the default preference[1], is installed instead
> of the one from other mirrors.
> 
> This is not always the case, sometimes there is no such race and the
> packages from DISTRO_APT_SOURCES mirrors are installed.
> 
> This makes buildchroot non-deterministic and affects caching.

Just to make the scenario even clearer: This is about packages needed
for the buildchroot-target which happen to be rebuilt by the
buildchroot-host (the other way around is not possible), practically
essential packages that - for whatever reason - requires a rebuild.

> 
> One solution is to provide a mechanism to set preferences for
> buildchroot. This way, the user would have flexibility to control the
> mirrors from which certain packages are installed.
> 
> This RFC introduces 2 new variables
> 1. BUILDCHROOT_HOST_APT_PREFERENCES
> 2. BUILDCHROOT_TARGET_APT_PREFERENCES
> to set the preferences for buildchroot-host/target respectively.

What is the difference to HOST_DISTRO_APT_PREFERENCES vs.
DISTRO_APT_PREFERENCES? That must be made clear I think.

Jan

> 
> Ofcourse, there might be more ways to solve this problem. Happy to
> discuss them here.
> 
> Thanks,
> Vijai Kumar K
> 
> [1]: https://github.com/ilbers/isar/blob/919fc995fc8ecb675f8bf639ee8628f90728b1ea/meta/classes/rootfs.bbclass#L90
> 
> Vijai Kumar K (2):
>   meta: Move aggregate_files to base class
>   buildchroot: Add provision to specify apt preferences
> 
>  meta/classes/base.bbclass                     | 10 +++++++
>  .../isar-bootstrap/isar-bootstrap.inc         | 10 -------
>  .../buildchroot/buildchroot.inc               | 30 +++++++++++++++++++
>  3 files changed, 40 insertions(+), 10 deletions(-)
> 

-- 
Siemens AG, T RDA IOT
Corporate Competence Center Embedded Linux