[PATCH 0/5] Improve buildstats functionality

[PATCH 0/5] Improve buildstats functionality

[Uladzimir Bely]

This series improves and enables by default buildstats in Isar.

Patches 2 and 3 are also sent to OE upstream in order to avoid forking
the code in Isar.

Uladzimir Bely (5):
  buildstats: Sync code base with openembedded-core 2022-04.15
  buildstats: consider multiconfigs when collecting statistics.
  buildstats: support of custom disk usage command
  meta/bitbake.conf: Use custom disk usage command for buildstats
  buildstats: Collect build statictics by default

 meta-isar/conf/local.conf.sample                 |  2 +-
 meta-test/conf/local.conf.sample                 |  3 +++
 meta/classes/buildstats.bbclass                  |  7 ++++++-
 meta/conf/bitbake.conf                           |  2 ++
 meta/lib/buildstats.py                           |  4 ++--
 scripts/pybootchartgui/pybootchartgui/draw.py    | 10 +++++++++-
 scripts/pybootchartgui/pybootchartgui/parsing.py |  2 +-
 7 files changed, 24 insertions(+), 6 deletions(-)

-- 
2.20.1

[PATCH 1/5] buildstats: Sync code base with openembedded-core 2022-04.15

[Uladzimir Bely]

This just syncs buildstats related stuff with recent tag in LTS
'kirkstone' branch of OE core.

Corresponds to OE commit eea685e1caaf.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/lib/buildstats.py                           |  4 ++--
 scripts/pybootchartgui/pybootchartgui/draw.py    | 10 +++++++++-
 scripts/pybootchartgui/pybootchartgui/parsing.py |  2 +-
 3 files changed, 12 insertions(+), 4 deletions(-)

diff --git a/meta/lib/buildstats.py b/meta/lib/buildstats.py
index 8627ed3c..c52b6c3b 100644
--- a/meta/lib/buildstats.py
+++ b/meta/lib/buildstats.py
@@ -43,8 +43,8 @@ class SystemStats:
         # depends on the heartbeat event, which fires less often.
         self.min_seconds = 1
 
-        self.meminfo_regex = re.compile(b'^(MemTotal|MemFree|Buffers|Cached|SwapTotal|SwapFree):\s*(\d+)')
-        self.diskstats_regex = re.compile(b'^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')
+        self.meminfo_regex = re.compile(rb'^(MemTotal|MemFree|Buffers|Cached|SwapTotal|SwapFree):\s*(\d+)')
+        self.diskstats_regex = re.compile(rb'^([hsv]d.|mtdblock\d|mmcblk\d|cciss/c\d+d\d+.*)$')
         self.diskstats_ltime = None
         self.diskstats_data = None
         self.stat_ltimes = None
diff --git a/scripts/pybootchartgui/pybootchartgui/draw.py b/scripts/pybootchartgui/pybootchartgui/draw.py
index 29eb7505..707e7fe4 100644
--- a/scripts/pybootchartgui/pybootchartgui/draw.py
+++ b/scripts/pybootchartgui/pybootchartgui/draw.py
@@ -267,7 +267,10 @@ def draw_chart(ctx, color, fill, chart_bounds, data, proc_tree, data_range):
     # avoid divide by zero
     if max_y == 0:
         max_y = 1.0
-    xscale = float (chart_bounds[2]) / (max_x - x_shift)
+    if (max_x - x_shift):
+        xscale = float (chart_bounds[2]) / (max_x - x_shift)
+    else:
+        xscale = float (chart_bounds[2])
     # If data_range is given, scale the chart so that the value range in
     # data_range matches the chart bounds exactly.
     # Otherwise, scale so that the actual data matches the chart bounds.
@@ -555,6 +558,11 @@ def render_processes_chart(ctx, options, trace, curr_y, w, h, sec_w):
             draw_rect(ctx, PROC_BORDER_COLOR, (x, y, w, proc_h))
 
             draw_label_in_box(ctx, PROC_TEXT_COLOR, process, x, y + proc_h - 4, w, proc_h)
+
+            # Show elapsed time for each task
+            elapsed_time = f"{trace.processes[process][1] - start}s"
+            draw_text(ctx, elapsed_time, PROC_TEXT_COLOR, x + w + 4, y + proc_h - 4)
+
             y = y + proc_h
 
     return curr_y
diff --git a/scripts/pybootchartgui/pybootchartgui/parsing.py b/scripts/pybootchartgui/pybootchartgui/parsing.py
index b42dac6b..9d6787ec 100644
--- a/scripts/pybootchartgui/pybootchartgui/parsing.py
+++ b/scripts/pybootchartgui/pybootchartgui/parsing.py
@@ -128,7 +128,7 @@ class Trace:
     def compile(self, writer):
 
         def find_parent_id_for(pid):
-            if pid is 0:
+            if pid == 0:
                 return 0
             ppid = self.parent_map.get(pid)
             if ppid:
-- 
2.20.1

[PATCH 2/5] buildstats: consider multiconfigs when collecting statistics.

[Uladzimir Bely]

When multiconfigs are used, statistics for the same package name but
from the different multiconfigs is collected into the same file.

This causes incorrect charts generated with pybootchartgui, when
only the most recent part of statistics is shown.

This patch adds custom multiconfig prefix to the file names that
hold statistics.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/buildstats.bbclass | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
index 0de60520..b417929b 100644
--- a/meta/classes/buildstats.bbclass
+++ b/meta/classes/buildstats.bbclass
@@ -185,6 +185,8 @@ python run_buildstats () {
     if bn is not None:
         bsdir = os.path.join(d.getVar('BUILDSTATS_BASE'), bn)
         taskdir = os.path.join(bsdir, d.getVar('PF'))
+        if d.getVar('BB_CURRENT_MC') != 'default':
+            taskdir = os.path.join(bsdir, d.getVar('BB_CURRENT_MC') + '_' + d.getVar('PF'))
         if isinstance(e, bb.event.HeartbeatEvent) and bb.utils.to_boolean(d.getVar("BB_LOG_HOST_STAT_ON_INTERVAL")):
             bb.utils.mkdirhier(bsdir)
             write_host_data(os.path.join(bsdir, "host_stats_interval"), e, d, "interval")
-- 
2.20.1

[PATCH 3/5] buildstats: support of custom disk usage command

[Uladzimir Bely]

This helps to make buildstats code usage easier in third-party projects
like Isar (https://github.com/ilbers/isar/). In Isar, rootfs is created
with 'sudo' privileges and some subpaths like '/proc' may be mounted at
build time. So, using "du -sh" on rootfs may produces multiple
'Permission denied' warnings.

Customizable disk usage command allows to deal with these issues
(e.g., by adding "-x" option or "sudo" can be added).

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/classes/buildstats.bbclass | 5 ++++-
 1 file changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta/classes/buildstats.bbclass b/meta/classes/buildstats.bbclass
index b417929b..94e48bfc 100644
--- a/meta/classes/buildstats.bbclass
+++ b/meta/classes/buildstats.bbclass
@@ -8,6 +8,8 @@ BUILDSTATS_BASE = "${TMPDIR}/buildstats/"
 #
 ################################################################################
 
+BUILDSTATS_DISK_USAGE_CMD ??= "du -sh"
+
 def get_buildprocess_cputime(pid):
     with open("/proc/%d/stat" % pid, "r") as f:
         fields = f.readline().rstrip().split()
@@ -246,7 +248,8 @@ python run_buildstats () {
                 rootfs = d.getVar('IMAGE_ROOTFS')
                 if os.path.isdir(rootfs):
                     try:
-                        rootfs_size = subprocess.check_output(["du", "-sh", rootfs],
+                        rootfs_size = subprocess.check_output(
+                                d.getVar('BUILDSTATS_DISK_USAGE_CMD').split() + [rootfs],
                                 stderr=subprocess.STDOUT).decode('utf-8')
                         f.write("Uncompressed Rootfs size: %s" % rootfs_size)
                     except subprocess.CalledProcessError as err:
-- 
2.20.1

[PATCH 4/5] meta/bitbake.conf: Use custom disk usage command for buildstats

[Uladzimir Bely]

In Isar, default "du -sh" disk usage command produces multiple
"Permission denied" warnings. Contrary to OE, Isar uses "sudo"
for rootfs creation and some path may be not available.

Also, some subpaths of rootfs may be mounted during statistics
collecting, so we have to limit "du" by one filesystem.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta/conf/bitbake.conf | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/meta/conf/bitbake.conf b/meta/conf/bitbake.conf
index 80dc01c7..a90edc4b 100644
--- a/meta/conf/bitbake.conf
+++ b/meta/conf/bitbake.conf
@@ -159,6 +159,8 @@ INHERIT += "isar-events sstate"
 # Buildstats requires IMAGE_ROOTFS to be always defined
 IMAGE_ROOTFS ??= "${WORKDIR}/rootfs"
 INHERIT += "${@'buildstats' if d.getVar('USE_BUILDSTATS') == '1' else ''}"
+# Use custom disk usage command to avoid "Permission denied" warnings
+BUILDSTATS_DISK_USAGE_CMD = "sudo du -shx"
 
 # Default values for ccache
 USE_CCACHE ??= "0"
-- 
2.20.1

[PATCH 5/5] buildstats: Collect build statictics by default

[Uladzimir Bely]

This enables collecting statistics during build.

Signed-off-by: Uladzimir Bely <ubely@ilbers.de>
---
 meta-isar/conf/local.conf.sample | 2 +-
 meta-test/conf/local.conf.sample | 3 +++
 2 files changed, 4 insertions(+), 1 deletion(-)

diff --git a/meta-isar/conf/local.conf.sample b/meta-isar/conf/local.conf.sample
index 17455015..eb539f51 100644
--- a/meta-isar/conf/local.conf.sample
+++ b/meta-isar/conf/local.conf.sample
@@ -197,7 +197,7 @@ USER_isar[password] = "isar"
 USER_isar[flags] += "clear-text-password"
 
 # Use buildstats by default
-#USE_BUILDSTATS = "1"
+USE_BUILDSTATS = "1"
 
 # Uncomment the below line to debug WIC.
 # WIC_CREATE_EXTRA_ARGS += "-D"
diff --git a/meta-test/conf/local.conf.sample b/meta-test/conf/local.conf.sample
index f692f533..f8e42612 100644
--- a/meta-test/conf/local.conf.sample
+++ b/meta-test/conf/local.conf.sample
@@ -48,3 +48,6 @@ USER_isar[comment] = "My isar user"
 USER_isar[flags] = "system create-home"
 USER_isar[password] = "isar"
 USER_isar[flags] += "clear-text-password"
+
+# Use buildstats by default
+USE_BUILDSTATS = "1"
-- 
2.20.1