[PATCH v1 0/1] Use oe.utils.directory_size() instead of du for calculating rootfs size

[PATCH v1 0/1] Use oe.utils.directory_size() instead of du for calculating rootfs size

[Adithya Balakumar]

Currently the "du" command is used to calculate the rootfs size, and sometimes the
reported size of the rootfs by du varies across multiple builds (usually by a few kb).
But the rootfs contents are reproducible.

Although this does not affect any functionality, but it does affect reproducibility
of artifacts that rely on the calculated rootfs size.

There are 2 scenarios (that I noticed) where this is affects reproducibility:
1. In the case where no size parameter is set for a rootfs partition in the wks file,
   wic uses the calculated rootfs size.
2. ext4 filesystem images with the IMAGE_CMD of isar, relies on rootfs size to create image.

Some of the files/directories in the rootfs has different number of blocks allocated.
The files/directories affected are not always the same across builds.

Using the "du" command to measure the size of the rootfs directory in such a case, yields
different values for the ROOTFS_SIZE.

This patch introduces the use of an alternate function from oe.utils to calculate the
directory size. OE-Core uses the same function to calculate the directory size (See [1])

Based on multiple build runs, usage of this function seems to give consistent size results.

Some details on the issue is mentioned in the below thread in the ISAR mailing list (See [2])

[1] https://github.com/openembedded/openembedded-core/commit/d8f1f3a6b024a2ae6631d1ce25421e8d94b69a12
[2] https://groups.google.com/g/isar-users/c/Ll7t4G41Lfo/m/gvxR4hYBAwAJ

Adithya Balakumar (1):
  image.bbclass: use oe.utils.directory_size() instead of du

 meta/classes/image.bbclass | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

-- 
2.39.5


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241129062827.527177-1-Adithya.Balakumar%40toshiba-tsip.com.

[PATCH v1 1/1] image.bbclass: use oe.utils.directory_size() instead of du

[Adithya Balakumar]

Currently the "du" command is used to calculate the rootfs size, and
sometimes the reported size of the rootfs by "du" varies across multiple
builds (usually by a few kb). But the rootfs contents are reproducible.

Although this does not affect any functionality, but it does affect
reproducibility of artifacts that rely on the calculated rootfs size.

Some of the files/directories in the rootfs has different number of
blocks allocated and "du" estimates the directory size based on the
allocated blocks. The files/directories affected are not always the
same across builds.

OE-Core uses a custom function to calculate directory size (See [1])
and this patch introduces the use of that function from oe.utils to
calculate the directory size.

[1] https://github.com/openembedded/openembedded-core/commit/d8f1f3a6b024a2ae6631d1ce25421e8d94b69a12

Signed-off-by: Adithya Balakumar <Adithya.Balakumar@toshiba-tsip.com>
---
 meta/classes/image.bbclass | 7 ++-----
 1 file changed, 2 insertions(+), 5 deletions(-)

diff --git a/meta/classes/image.bbclass b/meta/classes/image.bbclass
index 982523b0..3e70fe83 100644
--- a/meta/classes/image.bbclass
+++ b/meta/classes/image.bbclass
@@ -85,13 +85,10 @@ inherit image-account-extension
 ROOTFS_EXTRA ?= "64"
 
 def get_rootfs_size(d):
-    import subprocess
+    import subprocess, oe.utils
     rootfs_extra = int(d.getVar("ROOTFS_EXTRA"))
 
-    output = subprocess.check_output(
-        ["sudo", "du", "-xs", "--block-size=1k", d.getVar("IMAGE_ROOTFS")]
-    )
-    base_size = int(output.split()[0])
+    base_size = int(oe.utils.directory_size(d.getVar("IMAGE_ROOTFS")) / 1024)
 
     return base_size + rootfs_extra * 1024
 
-- 
2.39.5


-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/20241129062827.527177-2-Adithya.Balakumar%40toshiba-tsip.com.

Re: [PATCH v1 0/1] Use oe.utils.directory_size() instead of du for calculating rootfs size

[Uladzimir Bely]

On Fri, 2024-11-29 at 11:58 +0530, Adithya Balakumar wrote:
> Currently the "du" command is used to calculate the rootfs size, and
> sometimes the
> reported size of the rootfs by du varies across multiple builds
> (usually by a few kb).
> But the rootfs contents are reproducible.
> 
> Although this does not affect any functionality, but it does affect
> reproducibility
> of artifacts that rely on the calculated rootfs size.
> 
> There are 2 scenarios (that I noticed) where this is affects
> reproducibility:
> 1. In the case where no size parameter is set for a rootfs partition
> in the wks file,
>    wic uses the calculated rootfs size.
> 2. ext4 filesystem images with the IMAGE_CMD of isar, relies on
> rootfs size to create image.
> 
> Some of the files/directories in the rootfs has different number of
> blocks allocated.
> The files/directories affected are not always the same across builds.
> 
> Using the "du" command to measure the size of the rootfs directory in
> such a case, yields
> different values for the ROOTFS_SIZE.
> 
> This patch introduces the use of an alternate function from oe.utils
> to calculate the
> directory size. OE-Core uses the same function to calculate the
> directory size (See [1])
> 
> Based on multiple build runs, usage of this function seems to give
> consistent size results.
> 
> Some details on the issue is mentioned in the below thread in the
> ISAR mailing list (See [2])
> 
> [1]
> https://github.com/openembedded/openembedded-core/commit/d8f1f3a6b024a2ae6631d1ce25421e8d94b69a12
> [2]
> https://groups.google.com/g/isar-users/c/Ll7t4G41Lfo/m/gvxR4hYBAwAJ
> 
> Adithya Balakumar (1):
>   image.bbclass: use oe.utils.directory_size() instead of du
> 
>  meta/classes/image.bbclass | 7 ++-----
>  1 file changed, 2 insertions(+), 5 deletions(-)
> 
> -- 
> 2.39.5
> 
> 

Applied to next, thanks.

-- 
Best regards,
Uladzimir.



-- 
You received this message because you are subscribed to the Google Groups "isar-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to isar-users+unsubscribe@googlegroups.com.
To view this discussion visit https://groups.google.com/d/msgid/isar-users/ea58395cb344d6f16d968e273d1707b20f868e2a.camel%40ilbers.de.